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FOREWORD 



Educational outcomes have coinnianded increased atterition 
in recent years as the focus o£ evaluation has shifted from 
processes to products and Impact, hrm vocational education 
programs achieving their goals? What are the side effects 
of vocational education? What are the tangible payoffs to 
individuals and society from resources invested in vocational 
education? 

In this report of The National Center's study "EKamln- 
ing Vocational Education Outcomes and Their Correlates^" slg^ 
nificant progress is demonstrated in understanding the state 
of the evaluation art and expanding our awarenesa of the 
nature and diversity of educational outcomes. An Annotated 
Bibliography of Outcome Studies (Attachment "A" of this re^ 
port) documents some of the practical probleins confronted in 
conducting empirical research and evaluation on the outcomea 
of vocational education^ A Thesaurus of Outcome Questions 
has been developed listing 228 questions about the possible 
outcomes of vocational education for individuals and society 
(see Attachment ''B" in the report) * 

These two products^ along with an essay that provides 
needed perspective on the theory and practice of outcoines 
analysis (Attachment "C")r represent significant contribu- 
tions to the state of the art in vocational education eval-^ 
uation. The National Center and other members of the voca^ 
tional education community can use these prfiducts to help 
answer the questions i What particular outcomeB are appro- 
priate for evaluating vocational education programs? How 
can these outcomes be operationali^ed for purposes of eval- 
uation? A continuing research and development effort %^ill 
occur during the second year of this project? along with 
other specific endeavors alluded to in the essay ? which will 
help improve the quality^ credibility^ and tmpaat of evalua- 
tion efforts in vocational education . 

On behalf of The National Center ^ I want to eKpress 
appreciation to the Bureau of Occupational and Adult Educa= 
tion/ U. S* Office of Education, for sponsoring this study 
and to a number of individuals who contributed importantly 
to the success of this project. These include the distin- 
guished members of a National Workshop Panel on Vocational 
Education Outcomes/ including representatives from business r 
labor and educations Walter G. Davis ^ Mary Ellen Hillaire^ 
Addison S. HobbS/ 0. Louise Lothspelch, F, D, Mack, Gerald 
Q* Miller, Philip L. Smith/ Jerry C, Olson, Wallis E* Per- 
eira, Clio S# Reinwald, Annell L. Simcoe/ and J. Robert 
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Warmbrad. (Sea chapter II for information on professional 
affiliation and geographic location, of panel menibers.) 

We are alao grateful to Dr. Michael D* Hock Worth- 
ington, Ohio; Dr, Douglas Sjogren cf Colorado State Univer-- 
sityi and Dr* Michael Scriven of the University of San Fran- 
cisco / all of whom served as consultants for the study i and 
to tha five members of The National Center's Evaluation Tech 
nical Advisory Panels Miss Carol B. Aslanian, College En- 
trance Examination Board r Dr. George C. Copa , University of 
Minnesota^ Dr. Donald W. Drewes^ Conserva Inc.; Dr, Ruth P, 
Hughes, Iowa State University i and Dr. Daniel L, Stuff lebeam 
Western Michigan University » 

The report was prepared for The National Center by 
Robert L. Darcy, Kathleen A. Bolland^ and Joanne Farley , 
with the assistance of Carolyn Taylor. 



Robert E. Taylor 
Executive Director 
The Nati.onal Center for Research 
in Vocational Education 
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PREFACE 



This report includes a sumrnary of research and 
developnient mctivities^ findings, and prodvicts resulting 
from the first year of The National Center's applied 
R Si D study "Examining Vocational Education OutGomes 
and Their Correlates." 

Overall goals for the long^terr ntudy (initially 
proposed aa a five-'year effort) are to (A) identify/ 
operationalisG . and test a pool of outcnras for evaluate 
ing voGationri]. education and (B) identify program 
characteristicB And intervening factors hinhly corre- 
lated with those outcomes. During the first year 
(January 1 STC-n^nnuary 1979) / effort v^as flocused on three 
major objectivoDr (1) compiling a cor^nrehensive list 
of vocational education outcomes; (2) conducting a 
focused revie^^ of evaluative studies; and (3) developing 
a framework for analysing outcomes and the role they 
play in evaluation* 

RefLectin'^^ these activities^ the report includeB 
a brief discuBsion of objectives^ rationale and badc-^ 
ground for the study (Chapter I); a sunjnary of project 
activltiee and accomplishments (Chapter II); suggested 
applications of the R&D results and reconmendations 
for continuing study (Chapter 111) r an annotated bibliography 
of vooatioiial education outcome studies (Attactment "A"); 
a thesaurus o:^ vocational education outcone questions 
(Attachnient "B ) ; and an essay that provides perspective 
on outcomes evaluation (Attachment "C'^)*. Appended to 
the bibltography of outcome studies (Attachment "A") is 
a categorized annotated bibliography of literature 
closely related to the study of vocational education 
evaluation* • 

While the three attachments are nroperly viewed 
as integral^ nutually reinforcing components of a single 
report, they Also may be used as separate, self-contained 
documents , 

The basic research question addressed during the 
first year of thin study was; What particular outcomes 
are appropriate for use as criteria to evaluate voca^ 
tional education programs? We believe we have made 
significant progress toward answering this question. 
However^ this report does not contain a definitive list 
of 6^ 8r or 10 specific outcomes or 'outcome command- 
ments" for vocational education evaluators. Nor do we 
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Lay out B Btep*-by-step procedure for procHicing such a 
List* I^deed^ v/e have found no simple solutions to 
the methoaolo«^ical and standard-of-value dileirimas in^ 
educatloml evaluation^ only the possibility for making 
intelligent cholceB, We hope this ppportunity mil be 
fully aKpLoitOf^, in the months and years ahead. 

* * * 

In his foreword, the eKecutive director of The 
National Center appropriately acknov/ledged contribu- 
tiona made to this study by a number of individuals, 
I msh to adc: to that list the nanes of Gary Milogarek^ 
Trisha Arthiir, and Jeanette HcConaughy, who helped with 
various aspecto of the study; Dr. Jo ^nn Fteigar^^ 
Dr. David Pucel, Ms. Lynne Peterson^ and Dr. William 

Hull^ who reviewed a draft of the re-;50rt; and 
Paulina Jacobo . whose professionalism^ creativity / and 
secretarial skiLls contributed so importantly to the 
success of thn pro j sot. Finally, Kathlaen Bolland and 
Joanne Farley join me in thanking our colleagues in 
the Evaluation and Policy Division and othor mambers 
of The National Center staff for their cooperation and 
many helpful suggestions during the past year* 



January 197 9 Project Director 
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EXECUTIVE SUMMARY 



This report describes activities and results of an an^- 
plied research and development project examining vocational 
education outcomeB. Three products were developed: (1) an 
evaluative annotated bibliography of vocational education 
outcome studies - (2) a theBaurus of vocational education out' - 
cone questions I and (3) an essay on conceptual and practical 
iBSues in outcomes evaluation. These products will help 
answer a reBearch question of fundamental importance to voca- 
tional education^ namely i What particular outcomes are 
appropriate for use as criteria to evaluate vocational edu- 
cation programB? 

The report is addressed primarily to the research,, devel" 
opnientr and evaluation communities in vocational education 
and the broader fields of education and training. Much of 
the information will also be inmediately useful to policy 
analysts and educational leaders. The report is not intended 
as a handbook for practitioners. The impact of the report 
on vocational education audiences^ including special needs 
subpopulations , will be more direct and assessable after con-^- 
pletion of the second year of R ^ D activities. 

One of the findings of the study was that discussions ^ 
analyses^ and reports dealing with vocational education out-- 
comes are often ambiguous. The term "outcome" is frequently 
used interchangeably with "goal" or **ob jectlves , " and opera- 
tional definitions (empirical indicators) are unclear or in-- 
consistent. In this report^ vocational education outcomes 
are defined as "the consequences of vocational programs." 
ThaBe consequences may affect particular individuals ^ specific 
institutional entities, or society as a whole* Outcomes . in- 
clude not just intended benefits but also side effects that 
may be direct or indirect, economic or noneconomic, favorable 
or unfavorable. In order to evaluate vocational programs 
with reepect to outcomes, it is necessary to state the out- 
comes clearly, specify empirical indicators (outcome measures) ^ 
make quantitative or qualitative observations, identify eval-- 
uation standards, and then interpret the findings. Evaluation 
results can be applied for purposes of program improvement, 
accountability, and policy-making , 

The range and diversity of vocational education outcones 
is much greater than typically discussed by policy"-makers and 
studied by evaluators* The report lists 228 outcome questions 
t^hich illustrate the wide variety of possible results of 
vocational programs for special-needs subpopulations , women, 
high school youth, postsecondary students, and adults. Out-- 
comes addressed in these questions range from skill acquisition 
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and job plac^nent to educational attainment, increased earn- 
ings^ and such nonaaonomic outaomes as leadership development 
and improved self*- image among disadvantaged students ^ The 
report draws from this comprehensive list of outcome questions 
a shorter list of thirty questions as well as a table identi-' 
fying ten outcome hypotheses to illustrate outcomes with high 
potential for further intensive study* 

The number of evaluation studies focusing on QUtcomes 
was much smaller than had been expected, and the range of 
outcomes investigated was even more limited. Numerous voca-* 
tional education studies were found to be either needs assess- 
ments or descriptions of student characteristics/ programs 
and processes, resources, and educational context. Apparently 
little conceptual and empirical work has been done on voca- 
tional education outcomes for women and special needs sub- 
populations. With respect to the thirty-one empirical outcome 
studies included in the Annotated Bibliography, a number of 
methodological and data-related shortcomings seriously limit 
the validity and generalizability of the findings. It is 
therefore concluded that empirical research has not documented 
a substantial body of knowledge concerning the outcomes of 
vocational education. Additional evaluation studies which 
repeat the noted shortcomings of design and methodology will 
not add significantly to our accumulation of knowledge. 

Based on the findings of this study, it is recoimnended 
that researchers continue their intensive efforts to (1) 
specify a limited number of important outcomes that are both . 
appropriate and feasible to utilize in evaluating vocational 
programs and (2) operationalize these outcomes criteria by 
designing and field testing procedures that can be replicated 
by other evaluators* It is further recommended that the Bib- 
liography, Thesaurus, and Essay contained in the project re-- 
port be made available to the research, development, and 
evaluation coimnunity and to people in vocational education 
leadership and policy--making roles* Implementing these rec-^ 
ommendations will promote greater understanding of vocational 
education outcomes, help improve the quality of evaluation 
studies, and contribute to the development of a cumulative 
knowledge base that goes beyond strictly local, ad hoc, and 
transitory considerations. This in turn should lead to more 
effective and efficient vocational programs. 
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CHAPTER I 

OBJECTIVES, RATIONAIiE, AND BACKGROUND OF THE STUDY 



A. Objectives of the Study 

The basic intent of this study during its first 
year was to help answer the research question: What 
particular outcomes are appropriate for use as criteria to 
evaluate vocational education programs? Longer-term 
goals were listed in the "Propoeal for a National Center 
for Research in Vocational Education" (p* 1-192) as 
follows : 

• "to identify, operationalize , and test a pool 
of appropriate and feasible outcomes for 
assessing the effectiveness of vocational 
education* " 

• "to identify program characteristics and in^ 
tervening factors which are highly correlated 
with these outcomes," 

The study was proposed as a "five-year effort," 

In the Year-^One Technical Plan, four objectives were 
identified s (1) to develop a comprehensive/ categorized 
list of candidate outcomes for vocational education; 

(2) to screen and select outcomes for in-depth analysis; 

(3) to conduct a focused review of research and evaluative 
studies r informed opinions ^ and practice to analyze 
selected outcomes; (4) to synthesize and interpret in-- 
formation about selected outcomes/ with particular atten- 
tion to special need subpopulationa * " 

In a brochure developed several weeks into the project/ 
it was stated "This five-year study of vocational edu-- 
cation outcomes aims at identifying criteria that are appro- 
priate and feasible for evaluating vocational education pro-» 
grains* During the first year/ emphasis is placed on exam- 
ining a wide range of outcomes^ clarifying concepts/ and 
assessing in a preliminary way the extent to which agree- 
ment about .outcomes exists among relevant audiences." 

As work on the project progressed/ objectives acquired 
more operational meaning and clarity. Professional judg- 
ments were made concerning how the objectives might best be 
interpreted and effectively pursued. 
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B, BatiQiiaLe for the Study 



Increased understanding of educational outcomes is im- 
portant because vocational programs increasingly are being 
evaluated on the basis of impact rather than process* 
Under the provisions of the Educational Ajnendments of 1976, 
statea are required to evaluate prograni effectiveness on 
the basis of two specific outconie criteriai (a) student 
employinent in training-related occupations and (b) extent 
to which employers consider students to be well trained 
and prepared for employment. As expressed in the National 
Center proposal (p, 1-186), it would be "a narrow view" to 
assert that these are the only relevant indices, of the out-- 
comes of vocational education. Numerous other Qutcomes are 
identified and regarded by educators and others as being of 
rnajor importance to individual students and to society (note 
Figure 3 in Attachment "C" and see Attachment "B"). The 
Proposal (p. 190f) identified four central issues determin- 
ing the credibility of an evaluationi (1) clarity and 
specificity of outcomes r (2) acceptance of the outcomes by 
relevant audiences, (3) availability of tools/techniques 
that can sensitively measure the outcomes, and (4) "the 
feasibility (economic efficiency) of measuring outcomes. 
During Year One, the first two of these issues were specify 
ically addressed with results that are reported in Chapter 
II below. 

In CQncluding the statement of rationale for this 
project, the Proposal stressed "the need to , . * eKamine 
outcomes as central to the evaluation issues that voca- 
tional education will face in the inmediate future and for 
the years to come. The perennial charges and controversy 
about the appropriate scope of vocational education out^ 
comes will remain with us • . * , [The] need to become 
clearer about what we have learned to date on the matter of 
the outcomes is acute" (Proposal, p. 1^191)* 



Resea rch D esign and Procedures 

This project is identified under terms of the National 
Center eontract as an "independent Applied Research and 
Development Study*" Emphasis in Year One was on explora- 
tory research and the development of "instrumental" 
products (see Attachments "A", "B^" and "c") contributing 
to our expanding knowledge about vocational education out- 
comes. 

Planned procedures and activities for achieving the 
first-year objectives included (1) compiling a comprehensive 
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list of candidate outcomes from educational literature and 
informed persons, (2) conducting a focused review of liter- 
ature and practice ^ including outcomes relating to special 
needs lubpopulations i (3) formulating a strategy for having 
candidate outcomes reviewed by a technical advisory panel 
and a nontechnical task force, (4) developing alternative 
classification frameworks for categorizing outcomes g 
(S) identifying the characteriBtics of appropriate and 
feasible outcome rneasures, and (6) acreening and selecting 
a limited nuinber of outcomes (possibly 6^8} for further 
intensiva study as potential criteria to use in evaluating 
vocational education. Out of these efforts would come the 
following producta t 

m an evaluative annotated bibliography of vocational 
education outcomes literature 

• a compendium of outcome statements (ultimately a 
"thesaurus" of outcome questions) 

% an essay on the state of the outcomes -^evaluation 
art 

• a final report comprising the above products as 
well as summaries of background information^ pro^ 
ject activities/ and recommendations for further 
research and development 

A key activity planned to contribute to these procedures 
and products was a two-day workshop involving a group of 
distinguished consultants representing vocational education 
as well as other fields and interests* 



D> Evaluation Perspective and RSiD Implications 

During Year One ^ the study was not intended to evalu- 
ate vocational programs. Its aim was to expand our 
knowledge base concerning vocational education outcomes 
and outcomes evaluation so that the quality, relevance/ 
credibility , and impact of future evaluations will be en^ 
hanced * 

Specifically/ this initial phase of the longer--term 
applied R D study was intended to lay the foundation for 
subsequent conceptual and enipirical investigations that 
will help answer the two research questions addressed by 
this multi^year studyi 



1, What particular outcoities are appropriate for use 
as criteria to evaluate vocational education 
programs? 

2, What specific prQcedures and methodologies can 
be applied to tha selected outcome criteria to 
yield accurate, slgnificantr and reliable em* 
pirical knowledge concerning the effects of 
vocational education? 

Chapter II summarizes the project activities that ' 
were carried out to help build triis foundation and de^ 
scribes the accomplishnients during the first year of the 
National Center -s project "Examining Vocational Education 
Outcomes and Their Correlates." 



4 

1.6 



CHAPTER XI 

SUMMARY OF PROJECT ACTIVITIES AND ACCOMPLISHMENTS 



This chapter describes major project activities and 
accomplishitients I points out limitations of the study ^ and 
identifies staffs consultants, and other resources in--' 
volved in the project. For £ull details on the produats 
off this study, use Attachments "A," "B," and "C." 

A, Major Actiyitiea and Research Findings 

Staff effort during the year focused on six major 
activities i 

(1) formulating strategies and mobilising resources 
to accomplish project objectives 

(2) conducting a focused reviev^ of vocational aduca- 
tion outcoines literature and recording the re^ 
suits in the form of an annotated bibliography 

(3) compiling an extensive file of statements/ 
hypotheses describing the possible outcomes of 
vocational education and developing a thesaurus 
of outconie questions 

(4) developing a conceptual framework for addressing 
substantive concerns of the study 

(5) organizing and conductJ.ng a national workshop on 
vocational education outcomas 

(6) utilizing a variety of opportunities to assess 
the perceptions and preferences of vocational 
educators and others aonaeffning fch© appropriate^ 
ness# importance # and feasibility of particular 
outcomeB for further intensive study 

Continuing attention in the study was given to special 
needs subpopulations and the theine of sex fairness , Each 
of the areas of activity is briefly described balow* 

1^ Gearing Up , Early weeks of the first project year 
(January 16, 1978" to January 31, 1979) were devoted to re- 
viewing the project proposal (alluded to in Chapter I) , 
clevaloping technical plans ^ and mobilizing staff* Dis- 
oussions were held with National Center staffv docuinents r 
viewed, and efforts generally made to utilize to fullest 
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^dvantag© the bofly of acoumulated knov^ledg# in the area of 
outdoines analysis^ Section D and the AppendiK to this chap- 
ter provide information on staff and other resourcee used in 
the p3coj ect* 

2^ Review of Literature on Educattonal QutGOtnes ^^ A 
systennatio review of literature and praotioe relative to 
vocational education outcomes evaluation was initiated in 
April and continued into December 1978, 

Initially, it was expeoted that upvrards, of 100 outcome 
studies published since 196 8 would be available for review 
and soma method of screening and selection wouia have tO' be 
applied. What actually happened was that Initial extensive 
searching yielded only approKimately 30 reports that quali- 
fied as **Mpirlcal*' studies of "vocational education out-- 
comes*" All these studies have beeti included in the anno^ 
tated bibliography. During final stages of work on the bib* 
liography, approximately 15 additional appropriate docunients 
were identified but tin\e did not permit their inclusion* 

The eVa^luative annotated bibliography^ which was a 
major product of the literature review, has several parts. 
Technioal notes (Section II) preceding the ainnotations of 
empirical outcome studies provide readers a brief primer on 
me thoda logical Issues addressed in the evaluative annota-- 
tidhs * The epilogue (Sectioii IV) sumnari^es the substantive 
findings and methodological issues in the stiidles, Finallyy 
an appendix of key related literature is alsa included In 
Attachment "A*" This provides complete citations and brief 
annotations of a selection of publications relevant to a 
contextual understanding of vocational educaticn outcomes 
evaluation, 

3, Compilation of Outcome Statements , One of the 
earliest efforts in the project was to formulate working 
definitions of "vocational education outcomee" and "outcome 
stataments"^ for use as guides in compiling Outcome state- 
ments. Outcomes were broadly defined as "the consieguenoes 
of vocational programs," Outcome statements are simply, 
linguistic descriptions of those consequences. Typically , 
an outoome statement identifies what happens to whom (a per- 
son or a thing) and how this comes about. For example/ 
"Students who complete a vocational program acquire entry- 
level occupational skills," The what is skill acquisition 
(the outcome) . The to whom is studentB (the affected entity) 
and the how is by completing a vocational program (the edu- 
cational treatment) . In some instances an educational out- 
come statOTent also proviiaes information which answers ques- 
tions of when I where, and why. These concepts are treated 
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in detail in Attachments "B" and "C." where lists of outcoine 
questions (hypotheses) are also presented. 

The actual compilation began almost immediately and in- 
volvad an exploratory search of readily available literature 
as well as discussions with researchers and practitioners in 
vocational education. Each statOTient was recorded on a sep- 
arata file card with the source of the statement and an iden- 
tification number. Lists were periodically developed from 
the card file to illustrate outcome Btatements in designated 
categories (e*g.. individual vs. societal, employment-related 
vs, education-related) and to provide candidate outcomes for 
persons to rank-order in importance. 

Based on suggestions made by the Evaluation Technical 
Advisory Panel and a project consultar^t (see section below 
on Staff, Consultants, and Other Resources Used) , the de-^ 
cision was made to design a thesaurus of vocational educa- 
tion outcome questions to meet the need for a comprehen'- 
sive, categorized listing of outcome statements* The 
thesaurus appears in this report as Attachment "B." 

4, Conceptual and MethodQlog lcal Issues. h number of 
conceptual / definitional , and methodological Issues were 
confronted at the outset of this project. Among them were* 

— How to define "vocational education" 

— How to identify "a vocational student" 

—How to define "vocational education outcomes" 

{especially important in light of the propensity for 
practitioners and researchers to use the term as a 
synonym for goals/objectives) 

— ^How to differentiate between "outcomes" and "outcom;e 
measures" 

--How to conceptualise the task o*f evaluating vocational 
education on the basis of ^^utcoomas as opposed to 
other facets of the educatio^i enterprise (euch as 
processes, resources, goals, student characteristics, 
and school^and^conmiunity content) 

**-How tO' formulate "outcome stateinents" 

---How to classify or categorize outcomes and outcome 
statements 
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— Hov? to •'validate" particular outcomes in the sense of 
legitimising them as being important and appropriate 
for use as evaluation criteria 

--Ho^ to select and rank-order the outcomes that meet 
the tests of "validity" and feasibility (capability 
of being measured) 

Not all of these issues have been thoroughly analyzed* 
Attachments "B" and *'C," however, address the basic ques-^ 
tions and seek to provide the ideas and information that 
will be useful in subsequent analysis and ^ practioe. In par- 
ticular^ the essay "Examining Vocational Education Outcomes t 
Perspective on the State of the Art" (Attachment "C") is 
intended primarily as a contribution to the conceptual basis 
of outcomes analysis* 

5. National Workshop on Outcomes . An activity of 
critical importance , in the project was a two-day workshop 
held at The National Center on July 26--27^ 1978. Twelve 
individuals with, backgrounds r interests^ and rasponsibil-- 
itiee in diverse fields both within and outsida vocational 
education (see Table I) were carefully selected as an ad 
hoc task force to participate in the workshop. Efforts 
were made to assure that the panel was representative with 
respect to geographical location # race, seK, educational 
background, professional responilbilitiei , and other con^ 
siderations . 

The purpose of the workshop ^ as stated in the project 
plan, was "to help identify and validate outcomes appro- 
priate for further study." The rationale was to bring to-- 
gether a heterogeneous group of people interested in voca*- 
tional education and knowledgeable, as practitioners or 
scholars^ about the world of work. By interacting with 
staff and each other, the consultants were able to express 
their differing perceptions and values with respect to a 
broad range of education-related and work-related issues. 
A structured agenda provided for small-^group discussions 
(three or four consultants) , modified nominal group tech- 
nique, and other formal and informal interaction procedures, 

A preliminary "icebreaker" session was held the first 
morning to get the participants acquainted with each other 
and sensitive them to the perspectives, eKperiences, and^ 
values each individual was bringing to the workshop. This 
was follpwed by six "activity sessions" held during the^ 
intensive two-day workshop. These sessions involved indi- 
vidual, smail-group, and large-group written activities as 
well as discussions and efforts to achieve consensus. The 



Table I 



CONSULTANTS WHO PARTICIPATED IN THE NATIONAL 
WORKSHOP ON VOCATIONAL EDUCATION OUTCOMES 



WALTER DAVIS^ Director^ Department of Education, 
Amartcan Federation of Labor and Congress o£ 
Industrial Organizations, 
Washington , D * C * 

M^RY ELLEN HILLAIRE^ Instructor in Native American Studies, 
Evergreen State College* 
Olyinpia Washington 

ADDISON S* HOBBS, State Director of Vocational Education^ 
Michigan Department of Education, 
Lansing, r.lchigan 

O, LOUISE LOTHSPEICH, Director, Career Guidance and 

Counseling Project, Portland Public Schools, and 
Chair, Oregon State Advisory Council on Vocational 
Education * 
Portland, Oregon 

F. D* MACK, Teacher Educator, Health Occupations, Division 
of Vocational Education, Central State University. 
Edmonds Oklahoma 

GEM,LD Q. MILLER, Vice President for LaJbor and Industrial 
Relations, Sharon Stael Corporation, and Chairinani 
Employers National Job Service Improvement Conm\ittee, 
Sharon , Pennsylvania 

PHILIP L. SMITH, Assistant Professor of Education^ Faculty 
of Curriculum and Foundations, The Ohio State 
University. 
Columbus / Ohio 

JERRY C* OLSON, Superintendent, Pittsburgh Public Schools - 
Pittsburgh, Pennsylvania 

WALLIS E* PEREIRA, Director, Construction Education 

Program, Association of California Contractors, and 
MembaX/ Board of Directors of the Industry Education 
Council of California, 
Los AngeleSi California 



(CONTINUED ON THE FOLLOWING PAGE) 
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Table I (Continued) 



CLIO S. RE3NWALD, Direator, Home Economics and Hospitality 
Education^ Division of Career and Vocational Educa- 
. tlon/ A^rizona Department of Education. 
Phoanix/ Arizona 

ANNElIi L. SIMCOE, Chairperson, Department of Vocational- 
Technical Education I Graduate School of Education , 
RutgarB University. 
New BrianswicJc, New Jersey 

J. ROBERT mRMBROD, ProfeBSor of Agricultural Education 

and Past President of the American Vooational Educa^ 
tion Research Aesociation, The Ohio State University, 
Colunibus , Ohio 
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following topics were addressed i (1) identification of par» 
ticular voGational education outcomes; (2) categorization of 
outcomes; (3) clarification and specification of outcome 
statements F (4) rank-ordering of outcomes according to their 
importance and appropriateness for use as criteria to eval^ 
uate v^ocational education; (5) criteria for selecting and 
ranking outcomes; and (6) uses of wrkshop products and ex- 
periences in pursuing project objectiveB. 

Participation in this workshop was limited to the 12 
selected conBultants, members of the project staff, and a 
few observers. Because of the closed and intensive nature 
of the workshop, participants were extremely candid and ex^ 
plicit, both in expressing their own views and acknowledge" 
ing"^TtmaB of uncertainty and ambivalence, with respect to 
rank-ordering o£ candidate outcomes (see Table II) some 
agreement was indicated but no clear consensus emerged. 
The three outcome statements given the highest rankinge by 
the consultants are indicated in the table. It is noted 
with emphasis that these outcomes are not represented by 
The National Center or the project staff as in any way 
constituting a "validated" trio of vocational edueation 
outcomes* 

The rank^-ordering exercise^ with accompanying dis- 
cussion # was valuable because of what it revealed about 
differences in perceptions ^ expectations^ praferences^ 
underlying thinking^ and reactions to the differing views 
of others. Experience with this workshop session under-^ 
scored the complexity and subtlety of efforts to "validate" 
outcomes and reinforced the staff *s aommitment to avoidinq 
simplistic or authoritarian approaches to the selection of 
outcomes for further treatment. 

what came out of the workshop overall was a deeper 
appreciation of the conceptual and communication problems 
inherent in vocational education outcomes analysis. As one 
consultant stated s "Never again will I be so naive in my 
thinking and talking about educational outcomes," 

The workshop consultants eKpressed willingness and 
keen interest in convening again in 1979 for follow--up 
discuaslons of issues considered in the July 1978 workshop. 
They felt that after six months or more of reflection on 
the workshop eKperience, combined with an opportunity to 
review .the Year-One final report, they would be in an 
excellent position to contribute further to the project* 
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WORKSHOP ACTIVITY SESSION: 



RANKING OUTCOMES 



Name 



Handout for Activity Session #4 
RANKING OUTCOMES 



**************** 



Thursday^ July 27 



Listed below are 10 vocational eduoation outcome state^ 
ments, Your task is to rank the outcomes in terms of th^ir 
importance and appropriateness as indicators of the cense - 
quences of vocational programs . That Is^ assuming that 
valid^ accurate methods can be devised to observe and/or 
maasura the outcomes^ what is their rank-order of impor-^ 
tance for use in evaluation? Write number 1 by the most im- 
portaht item^ number 2 by the second most important r and 
so on through number 10, the least important. 

High school seniors enrolled in vocational programs 

~ are more satisfied with their school experience than 
are comparable students taking academic/general 
courses* 

Employers rate vocational students more satisfactory 
~ as employees than comparable academic/general students 
after six months on the job* 



Within six months of program completion, students ob- 
tained jobs related to their occupational training. 

High school dropout rates are lower for vocational 
students than comparable students enrolled in acaderaic/ 
general curricula* 

Two years after program completion, vocational students 
were more satisfied with their jobs than were graduates 
of academic/general programs holding similar jobs* 

Parents of high school seniors enrolled in vocational 
programs are more satisfied with the curriculum than 
parents of comparable students enrolled in academic/ 
general programs. 



Within six months of program completion^ students ob- 
tained satisfactory jobs, though not necessarily in 
the area of their occupational training. 

{CONTINUED ON THE FOLLOWING PAGE) 
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Table I I ( continued ) 

Upon completion of a vocational program, students were 
certified as occupationally proficient and ready for 
entry--level employment.. 

Private sector employers rate high school vocational 
programs in their communities higher than academic/ 
general programs , 

One year after program completion , vocational students 
were earning higher hourly wages than comparable 
academic/general students holding similar jobs. 
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6. "Valiaation" Activities . Throughout the year^ con- : 
tihuing effortn v;ere made to "validate-' particular outcomes 

In the sense off ^determining their appropriateness r impor- 
tanae^ legitimacYr acceptance , credihility as evaluation 
criteria for voaational programs * DiscuBsions were held 
with numerous individuals concerning outcomes they considered 
important for use in evaluating vocational education pro- 
grOTis* OpinionB varied according to the professional iden- 
tity of the reoponder (e.g»^ vocational education practi- 
tioner ^ academic researcher^ private sector representatives of 
bualness or organized labor) ; situation in which the question 
was posed (e,g*^ small group, large groups substantial orieii'- 
tation, little or no explanation) i and the demographic/socio- 
aconomlc background of the respondent. 

No effort was made to employ a formal Delphi method or 
conduct scientifically controlled surveys. In part this 
reflected the staff's judgment that there would be little 
likelihood of obtaining useful data^ i.e., responses that 
would be representative/ well-informed, responsible^ valid, 
authentic, and meaningful. While this approach is not 
ruled out for the future, the staff is acutely sensitive to 
the pitfalls of such efforts to achieve consensus, including 
the risks inherent in arbitrarily designating an electorate 
and simply counting the votes cast. This approach of 
selecting the "who'' to render value judgments seems less 
efficacious than developing a sound procedure for how such 
judgments can be made. 

7 . Research Findings . Detailed findings from the first 
year of this applied R&D study of vocational education out- 
comes are presented ins 

# Attachment "A" — "Vocational Education Outcome 
Studies! An Evaluative Annotated Bibliography 
Including an AppandiK of Key Related Litera- 
ture;" in particular, see Section IV, which 
sunmiarizes substantive findings and methodol- 
ogical issues based on studies published dur- 
ing the past decade 

m Attachment "B" —"Thesaurus of Vocational Edu- 
cation Outcome Questions;" in particular see 
Section I - 

• Attachment "C"— "EKamining Vocational Education 
Outcomes I Perspective on the State of the Art" 

We have concluded that at this stage of the multi-year 
effort there is insufficient basis for announcing a limited 
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selection of particular outcomes for in-depth analysis. 
While this was considered during the planning and early 
implementation stages, the action appears inappropriate at 
this tinie and might tend prematurely to focus unwarranted 
attention on such outcomes. We would not want evaluatora 
to think that the selected outcomes were in some sense pre- 
scribed by The Hational Center as "the" criteria to use in 
conducting evaluation studies, or even as outcomes recommend 
ed for priority R & D study. It is important at this stage 
that we not exclude from further analysis other outcomes of 
high potential in terms of importance, feasibility, and ap- 
propriateness . 

Ideally, the selection of outcomes should be grounded 
in theory and supported by empirical research. If the voca- 
tional education community were "clearer on the matter of 
outcomes" (see Chapter I), a sound basis might now exist for 
selecting and operationalizing outcomes for use in evalua- 
tion. Our recent investigation, however, has confirmed what 
practitioners and scholars alike have been saying; namely, 
that a great deal of the evaluation done in vocational edu- 
cation has suffered from serious conceptual, methodological, 
and data shortcomings. Simply doing more of the same kind 
of research an^ evaluation would be unlikely to advance sig- 
nificantly our knowledge of the consequences of vocational 
education. He therefore are convinced that R&D activities 
of the type described in this report are necessary and can 
generate valuahle returns as new ways are charted for im- 
proved outcomes evaluation in the years ahead. 

Uses to be nade of first-year findings and a strategy 
for second-year research and development efforts regarding 
selection of outcomes for special analysis will be discussed 
in Chapter III. 



B. Research and nevelopment Products 
a7nd Pis se'mination Plans 

Four applied research and development products result- 
ed from the first year of The National Center's vocational 
education outcomes study i 

(1) "Vocational Iducation Outcomes Studies: An 
Evaluative Annotated Bibliography lncludi,ng 
an Aopendix of Key Related Literature" (in- 
cluded as Attachment "A" in this report) 

(2) "Thesaurus of Vocational Education Outcome 
Questions" (included as Attachment "B" in 
this report) 
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(3) "Examining Vocational Education Outcomes s 
Perspective on the State of the Art" (in- 
cluded as Attachment '*C" in this report) 

(4) VOCATIONAL EDUCATION OUTCOMES i Final Re- 
port on Year One of the R a D Projeat 
"Examining Vocational Education Outaomes 
and Their Correlatea" (including Attach- 
ments "A." "B," and "C") , 

Fifteen copies of the complete Final Report will be de 
livered to the Bureau o£ Occupational and Adult Education^ 
U* S. Office of Education. Additional Gopies of the full 
report will be printed for use by The National Center and 
for individuals who contributed directly to the study* : 
Arrangements for wider diseeinination of the report will be 
considered through The National Center's coet-recovery docu 
ment sales program. 

Copies of the three respective attachments—the 
Annotated Bibliography^ Thesaurus.^ and Eesay^-rnay also be 
disseminated through The National Center's cost-recovery 
sales program as separate publications if market analysis 
warrants this procedure. It is also possible that the 
Annotated Bibliography will be divided into two separate 
publications^ namely the Annotated Bibliography of Voca^ 
tional Outcome Studies and a distinct Bibliography of Voca- 
tional Education Key Related Literature, 

It should be noted that these documents were developed 
as "instrumental productSi" aimed at establishing a 
knowledge base for carrying forward the multi-year study 
as described in Chapter I. To the extent that these first- 
year products serve other immediate functions i they may be. 
regarded as side benefits. 



C, Other Accomplishments y interfaces ^ and Limitations of 
the Study 

1 , Specialized Human Capital , Beyond the tangible 
products listed m Section~B, the most significant project 
accomplishment during the first year was an accumulation of 
specialized human capital in the field of outcomes analysis 
Staff members developed R&D capabilities in the form of 
pro ject-^specif ic knowledge ^ skills , and attitudes which can 
be applied productively in the second year of the study. 



16 



Notable is The National Center's keener awareness of 
pitfalli in outcomes evaluation and heightened sensitivity 
to the need for lucid thinking and clear conmiunication if 
breakthroughs are to be made in this crucial area of voca- 
tional education research^ development^ and evaluation* It 
is important that we acknowledge what we do not know about 
outcomes as well as report and use knowledge that we do 
possess . 

2_. I nterfaces , Members of the project staff took ad-- 

vantage of numerous opportunities to interact with voca- 
tional education practitioners and researchers as well as 
people outside vocational education. Approximately 15 0 
copies of a project brochure published in May 1978 (see 
facsimile/ Figure 1) were distributed in response to re- 
quests for information or at the initiative of project 
staff as a basis for discussion. Individuals were asked to 
suggest outcomes they personally considered to be highly 
appropriate for use in evaluating vocational education. In 
some instances they were given a list of eight or ten or 
twelve outcome statements and asked either to rank them in 
importance (for use as evaluation criteria) or classify 
them as "Very Important "Important/" or "Minimally Im- 
portant." 

As reported in the section on "validation" efforts^ 
these informal polls raised more questions than they 
answei'ed. 

Among the groups with whom project staff interfaced 
were participants in a "Leadership Conference for Admin^ 
istrators of Vocational Education in Large Cities" 
(Columbus / Ohio, April 19) ; staff members of The National 
Center's Evaluation and Policy Division (luncheon seminar 
and other occasions) and of the R a D Division and the 
Information and Field Services Division; participants in 
the **National Conference on Outcome Measures for Vocational 
Education" (Louisville^ Kentucky^ August 16^18, 1978)* 
More limited interfaces occurred with individuals workincr . 
with educational outcomes at the U*S, Department of Labor 
(Office of Program Evaluation and Research^ Employment and 
Training AdminiBtration) ; Public Services Laboratory 
(Georgetown University) ; Agency for International Develops- 
ment; ^orld Bank; Midwest Economics Association; National 
Commission for Employment Policy) U*S* Department of Com- 
merce (Economic Development Administration) ; Library of 
Congress (Congressional Research Service) ; staff directors 
of Congressional subcommittees in the education and employ- 
ment areas; National Institute of Education; and other 
educational/ research , and human resource organizations. 



17 



The National 
Center for 

Research in 
Nfocatibnal 
Education 




Examining 

Vocational Education 
Outcomes and 
Their Correlates 




THE NATIONAL CENTER 



FOR RESEARCH IN VOCATIONAL EDUCATION 

THE OHIO SmTI UNIVtHSITV 
1960 KENNY ROAD • COLUMIUS OHIO I J?in 



Figure 1 



18 

30 




g Vocational Education 
s and Tlieir Correlates 




THE NATIONAL CENTER 
FOR RESEARCH IN VOCATIONAL EDUCATION 

THE OHIO STAIL UNlVgRSlTY 

nm KrNNY road' Columbus omio a.i.'ui 



The Nitional Canter for Reiaarch in Vocat'onal 
Education's mission is to increase the ability of 
dlvarsB agencias, institutions, and organizations to 
solve aducationdl problems ratating to Individual 
career planning, preparatlDn, and progrission, 
The National Canter fulfills Its miision by: 

• Ganarating knowledge through rasearch 

• Daveloping educatlpnal programs and 
products 

• Evaluating Individual prograrn needs and 
outcomes 

• Installing aducatlonal programs and 
products 

• Oparating information systams and 
services 

• Conducting leadarship devalopment and 
training programs 



Robert i, Taylor 
Exacutiva Director 



FOCUS 



This five-year study of vocational education 
outcomei aims at identifying criteria that 
are appropriate and feaiibSe for evaluating 
vocational education progranns. 

During the first year, emphaiii is placed on 
examining a wide range of outcomei, clari- 
fying concepts, arid asseising in a prerimi- 
nary way the extent to which agreement 
about outcomes exists among relevant 
audiences. 



The National Center is studying not only 
traditional goals, objiGtivei, compliance 
standards, and presumed benefits luch as 
job placement in training-related fields, 
but alio the broader consequences of voca- 
tional education programs for individual 
studenti and society. Special needs sub- 
populationi and sex equity-^two major 
themes identified by the National Center 
for particular emphasis— are addressed in 
this project. 

Beginning in the second year, a number of 
selected outcomes will be studied to datar- 
mine how they correlate with program 
characteristics and other factors. 



Figure 1 (Continued) 
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SPONSORSHIP 

The Study is sponsortd by the Suraau of 
Occupational and Adult Eduoation under 
tfie terms of tiie U.S, Office of Education 
contract with the National Centeri The 
Project is an Applied Reseirch and Develop- 
ment task being carried out by the National 
Center's Evaluation Division^ There are 
Important interfaces between this Project 
and a National Center study focusing on 
the Intffpretation of outcome measures 
(sponsored by the National Institute of 
Education). 

SPECIAL ACTIVITIES 



An activity of critical importance in the 
first year of the study is a workshop to be 
held at the National Center in July 1978 
involving a group of distinguished consult- 
ant representing vocational education as 
v\^ll as other fields and interests. 

This workshop task force will contribute 
ideas and Judgments concerning particular 
outcomes, ways of classifying outcomes, 
priorities, and issues related to developing 
increased agreement concerning vocational 
education outcomes. 

In addition, Project EVEO will benefit 
from review and suggestions provided by a 
technical advisory panel of evaluation 
specialists from outside the National Centen 



PRODUCTS 

First-year products will include: (1) a 
comprehensive categorized listing of voca- 
tionil education outcomes, (2) a elected 
list of approximately 8-8 outcomes vali- 
dated and screened for intensive further 
study, (3) an evaluative annotated bibli- 
ography of outcome studies^ (4) a state-of- 
the-art essay, and (5) a final report sum- 
marizing project activities and findings 
during the year and outlining plans for 
continuing restarch In the coming year. 



* # * 



For additional Information, write or call: 

Progrirn Information Sarvlct 
Tht National Center for Reseirch 

In Vocational Eduaatlon 
The Ohio State University 
1960 Kenny Road 
Columbus, Ohio 43210 
(614)486'3655 



Fiqure 1 (Continued) 
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within The National Center's Evaluation and Policy 
Division, members of the project staff endeavored to main'- 
tain especially close liaison with three related evaluation 
projects I (1) Providing Technical Evaluation Assistance to 
State Education Agencies; (2) Developing Evaluation Hand- 
books for Vocational Education; and (3) Increasing the 
Quality of Vocational Education Evaluation Reports* A 
brief report on the project was also presented at a meeting 
of The National Center's Advisory Council. 

3, Limitat ions of the Study . This report covers the 
first year of a multi-year research and development effort. 
Activities during this period included detailed planning^ 
start-up ^ and the development of products that will con- 
tribute to answering the research questions What particu-^ 
lar outcomes are appropriate for use as criteria to evalu- 
ate vocational education programs? Project plans call for 
generating useful^ if not definitive^ answers to the 
question by the end of the second year. 

At this stage in the R&D effort we are better equipped 
to conBider when and how to select from the large pool of 
outcomes those particular outcomes that show high potential 
on the basis of importance ^ appropriateness i and feasibility, 
Indeedf we have already done some preliminary selecting. 
Table II above lists 10 outcome statements/hypotheses drawn 
from the comprehensive pool for the purpose of eliciting 
evaluative responses from the ad hoc panel of representa- 
tives participating in the Mational Workshop on Vocational 
Education Outcomes. In Attachment "C^" Table I lists 30 
outcome questions selected from a larger pool to illustrate 
the diversity of outcome themes i empirical indicators, en-- 
tities affected, educational treatments , and the like. In 
sum, we have sharpened our understanding of outcomes, iden- 
tified a broad and diverse range, considered criteria for 
selecting from a larger pool of outcomes, and developed an 
awareness of the sensitive issues inherent in choosing and 
publicizing a limited set of outcomes to be used as evalua- 
tion critaria. We therefore have progressed about 35 percent 
to 4 5 percent of the way toward accomplishing the first of 
the two long-term goals established for this^ study (see p. 
1) . With respect to the second long-term goal, the focused 
review of outcome studies and other project activities have 
suggested a number of factors which may be correlated with 
particular outcomes. 



Continuing attention throughout the first year was 
given to the question of how vocational education outcomes 
relate to special needs subpopulations and the theme of sex 
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fairness. Attempts to identify empirical outcome studies 
foausing on epeoial-needs groups and sex-^equity themes met 
with disappointment. This is reflected in the paucity of 
such studies in the Annotated Bibliography of Vocational 
Outcome Studies (Attachment "A"). A few documents focusing 
0X1 special groups and sex equity themes have been included 
in the appendix attached to the Annotated Bibliography. How- 
aver^ these deal with aspects of vocational education eval- 
uation related to but not directly focusing on outcomes per 
se- This lack of information also made it infeasible to 
attempt a systematic discussion of these topics in the state-* 
of--the-art essay (Attachment "C"). 

Project staff met with more success in simply identify'- 
ing outcomes concerning special-^needs groups and sex-'equity 
themes. Many of these came from colleagues in The National 
Center who are working on projects in these priority areas. 
These outcomes are reflected in the Thesaurus of Vocational 
Education Outcome Questions (Attactaent "B"). 

A final limitation of the study is that it has not pro- 
duced a choice of a classification system for categorizing 
outcomes. Alternative class if ioation schemes have been re^ 
viewed and some remain under consideration. In the Thesaurus 
of Vocational Education Outcome Questions, we have adopted 
a classification framework based on practical considerations 
rather than on substantive or theoretical grounds. For dis- 
cussion of the compleK issues that accompany the task of de- 
veloping a classification framework, see Section I of Attach- 
ment "B," 



P. Project Staff , Consultants # and Other Resources 

Principal members of the project staff during the 
first year were: 

Robert L. Darcy, Project Director 

Kathleen A- Bolland, Program Associate 

Joanne Farley ^ Program Assistant 

Carolyn M. Taylor, Graduate Research Associate 

Pauline Jacobs, Secretary 

Brief resumes of the professional staff are appended to 
this chapter. 

Prior to his appointment (July 1978) as Associate 
Director of the Division of Evaluation and Policy^ Nerval 
L, McCaslin also served on the project staff; and Nancy 
Powell was half-time project secretary until Septamber 1978 • 
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In addition to the workshop consultants listed in 
Table I above, the following individuals performed consult^ 
ing services in support of the projects 

Dr- Michael D, Hock, Consultant in Human Services 
Planning and Evaluation/ Worthington, Ohio 

Dr, Douglas Sjogren, Professor of Education, Colorado 
State University / Fort Collina, Colorado 

Dr, Michael Scriven/ Director, Evaluation Institute, 
University of San Francisco, San Francisco ^ 
California 

and members of The National Center's Evaluation Technical 
Advisory Panel i 

Carol B. Aslanian, Associate Director^ College Entrance 
EKamination Board, New York City 

Dr. George C* Copa^ Professor of Vocational and 

Technical Education , University of Minnesota , 
Minneapolis 

Dr. Donald Dreures^ President, Conserva, Inc.# 
. Raleighi North Carolina 

Dr, Ruth P. Hughes^ Head, Home Economics Education, 
Iowa State University^ Ames 

Dr. Daniel L. Stuff lebeam. Director^ The Evaluation 
Center, Western Michigan University, Kalamaioo 

Other resources contributing to this study Include 
the physical facilities, research library, and personnel of 
The National Center and of The Ohio State University, 
Budget data are available from The National Center and the 
U.S* Office of Education, 
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Appendix to Chapter II 



BRIEF RESUMES OF PRINCIPAL STAFF 



ROBERT L. DARCY joined the staff of The National 
Center in January 1978 as Senior Research Specialist. A 
native of Illinois, he attended public schools in the 
Chicago suburbs and earned degrees in econoinics from Knox 
college (B.A,), Indiana University (M.A J , and the 
University of Colorado {Ph,D.), Dr. Darcy was Professor 
of Economics at Colorado State University (Fort Collins) 
from 1968 to 1975, where he also directed the Center for 
Economic Education. He earlier served on the faculties at 
Ohio University and Oregon State University and has con- 
sulted for the U*S, Department of Labor , Office of Educa- . 
tion, National institute of Education, The World Bank, and 
other organizations. He has contributed to books and 
journals in the field of human resource econoinics and 
economic education and is the author of a text for ■ 
secondary school students, Manpower and Economic Educatioh . 

KATHLEEN A. BOLLAND joined The National Center staff as 
a program associate in March 1978. A native of Michigan, 
she attended Pontiac Public Schools and earned her B.A. in 
psychology from Oakland University (Michigan) and her MtS. 
in human experiinental psychology and quantitative psychoid 
ogy at the nnivprBlty of Washington, where she was a re- 
search and teaching assistant in the Department of Psychol- 
ogy. She is currently working on a second master's degree 
at The Ohio State University, in educational research and 
evaluation, Ms. Bolland gained professional experience as 
a research associate in private industry and as a program 
evaluator, grants coordinator, and regional testing 
specialist for the State of Ohio Department of Administra^ 
tive Services. She has taught courses in the area of 
program evaluation for state employees* 



JOANNE FARLEY joined The National Center staff in 
July 1978 as a program assistant. At The Ohio State Uni-- 
versity (1974--78) , Ms. Parley served as a graduate re-- 
search assistant and teaching assistant for the Department 
of Political Science. She earned her M.A. in political 
science and has completed general examinations for the 
Ph.D., specializing in political theory and international 
relations. Her research skills include statistical and 
computer experience. A native of Tennessee, she earned 
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her B*A. in political science and history from Memphis 
State University. 

CAROLYN M- TAYLOR has been a graduate research asaoaiate 
in the Evaluation and Policy Divlsdon of The National 
Center since September 1978* A native of Alabaina^ she 
earned her B,S* in secretarial science at Alabama College 
for Women (Montevallo University) and M.A, at The Ohio State 
University ^ whera she is currently TOrking toward the Ph*D* 
in Vooational^Technical Education* Ms, Taylor attended 
the University of California ^ Berkeley; San Francisco 
State University; and Oregon State University. She holds a 
secondary and coiraiunity college teaching credential and an 
administrative services credential from the State of Cali^ 
fornia^ where she taught on the secondary level for ten 
years before coming to Ohio* 
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CHAPTER III 



R&D APPLICATIONS AND RECOMMENDATIONS 
FOR CONTrNUING "STUDY 



This chapter suggeste some immediate and longer run 
applications of the R a D findings and products resulting 
from the first year of a proposad multi^year study "EKamining 
vocational Education Outcomes and Their Correlates." Recom= 
mendations are then offered both for carrying this particular 
effort forward and for related efforta aimed at improving 
vocational education evaluation. 



A. Proposed Applications of R & D Results 

The principal results from the first-year effort are 
more properly characterized as development products--the 
bibliography, thesaurus, essay, and what was referred to in 
Chapter II as specialized human capital — ^rather than 
research findings or evaluation conclusions* These results 
have instrumental value as capital goods which can be used 
in further production, both in The National Center's ongoing 
studies and elsewhere. The production of knowledge in the 
area of educational outcomes can, in turn, help improve 
vocational education policy, program design, program manage- 
ment, and ultimately educational performance and impact. 

1. Annotated Bibliog raphy of Outcome Studies > This 
product (Attachment "A") will be a major source of infor- 
mation on research design and methodology to be used in 
operationalizing selected outcomes for evaluating vocational 
programs. Strengths and weaknesses of past studies (see 
Attachment "A," especially the Introduction, Technical 
Notes, and Epilogue) will help guide The National Center 
staff in designing evaluation procedures. Sources identified 
in the Appendix of Key Related Literature can be more fully 
utilised in performing the in-^depth analysis of a more 
limited range of outcome' issues* 

For eKtarnal uses, the Annotated Bibliography of Out- 
come Studies represents a unique compendiOTt of recent 
empirical research and evaluation work in vocational education. 
While not exhaustive, the list of studies provides a cross-- 
section of what might be termed good, bad, and indifferent 
evaluative research. To a limited extent it fills the need^ 
described by a past president of the American Vocational 
Education Research Association, who called for a detailed 
review of the methodology and substance of evaluation studies. 



The AppendiK of Key Related Literature could prove of great 
value in helping orient newcomers to vocational educatiori 
research and evaluation^ as well as educational policy anal*^ 
ysta and others* While far from exhaustive^ the listing 
nevertheless identifies significant data basest review and 
synthesis papers , empirical studies ^ methodological pieces / 
and publications reflecting a variety of world-of-work per^ 
spectives, 

2* Thesaurus of Outcome Questions * The Thesaurus 
(Attachment "B") provides in Section II a categorized listing 
of 228 vocational education outcome questions. From this 
extensive pool^ the project staff will draw a nimiber of 
outcome statements for in-depth study ^ development^ and 
pilot^testing as criteria to evaluate vocational programs. 
The conceptual analysis presented in Section I of the 
Thesaurus will be applied in carefully formulating the out-^ 
come hypotheses. 

Beyond imnediate project applications, it is anticipated 
that the Thesaurus will serve a number of functions in the 
hands of external users. The range and diversity of outcomes 
can help policy makers broaden and deepen their perspectives, 
suggest different priorities, and indicate possible new 
evaluative criteria for vocational education. Researchers 
can peruse the list of outcome questions as potential 
research topics, noting the subject index for special inter** 
ests such as sex fairness or the needs of disadvantaged 
students. The Thesaurus can also be used in developing a 
checklist of hypotheses that have or have not already been 
tested. 

3, E ssay on O utcomes Analysis . The Perspective on the 
State of the Art (Attachment "C") provides the project staff 
with a vocabulary and framework for analysis and coimnunica- 
tion about educational outcomes. The essay identifies a 
number of specific issues that must be addressed in opera^ 
tionaliiing particular outcomes for use in evaluation. It 
lists some important hypothesised outcomes, describes dlf-- 
farent types of outcome data, and distinguishes Miong the 
various objectives of educational evaluation. 

External readers of this product may find the discussion 
of concepts, models, and issues stimulating and useful as 
a point of reference for analysis and comnentary. Policy 
makers, vocational educators, and the general public may 
benefit from an expanded awareness of the broad range and 
diversity of educational outcomes. The suggested agenda for 
improving outcomes evaluation may help to crystallize atten* 
tion and coimnitment to finding much-needed answers to a host 
of key questions affecting the future of vocational education. 
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B, Reconmendatlons for Further R & D Efforts 



Through this project and related efforts, 1979 could 
prove to be a year of significant progress for The National 
Center in the area of outcomes analysis » Certainly there 
is a need for careful review and synthesis of the knowledge 
advances resulting from the Interpreting Outcome Measures 
study and the state-^of--the-art essayB (alluded to in footnote 
two of Attachment "C") as well as the first-year results 
emanating from this particular R S D study. The highest 
priority should be given to mining these products for the 
potentially valuable content they have to offer* It would be 
a tragic mistake to plunge ahead into new terrain to the 
neglect of recently produced knowledge not yet processed for 
use, 

1, Second-Year Continuation Plans , Emphasis in the 
second year will be on operationali^ing a limited pool of 
outcomes that show good potential for use in evaluating the 
consequences of vocational programs. One or more of the 
outcome measures will then be pilot-tested during the year 
to determine the clarity of the hypothesis ^ adequacy of data^ 
validity and reliability of instruments ^ and soundnass of 
evaluation design- The procedure will also include (a) an 
asaessment of the extent to which support exists among 
relevant audiences for the particular criterion and (b) a 
demonstration of how the evaluation results could be inter- 
preted^ generalized^ applied^ and disseminated. 

Results of the operationalizing and pilot-testing will 
be used to help validate particular outcomes as appropriate 
criteria for evaluating vocational programs. The validation 
process will entail soliciting reactions and comments from 
such audiences as vocational education policy makers, admin-- 
istrators, teachers, and students as well as interest groups 
such as special needs subpopulations , parents, employment 
and training practitioners, and local, state, and federal 
government agencies. Consideration will be given to re-- 
convening the panel of workshop consultants td review pro-^ 
gress made in the study and advise on the validation of par- 
ticular outcomes (see Chapter II, Section above). 

Beyond the Second Year , When the Year-Two operational- 
izing and pilot-testing of outcomes is completed, further 
R&D activities and evaluation applications will be con- 
tinued either as a separate project (as described in The 
National Center Proposal, p. 1-192) or in conjunction with 
a proposed Longitudinal Study of Former Vocational Education 
Students and proposed Case Study of Vocational Education 
Programs to Determine Correlates of Placement. 
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VOCATIONAL EDUCATION OUTCOMES 
(Pinal Report on Year One of the R&D Project "Examining 
Vocational Education Outcomes and Their Correlates") 
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PREFACE 



This annotated bibliography ie a product of a focused 
review of the literature and practica of evaluating ^oca* 
tional education outcomeB oonduated at the outset of a 
project designed to identify faaeible and appropriate mea- 
sures for; assessing the consequences of vocational educa^ 
tion programs* It was compiled primarily to provide the ■ 
vocational education coimnunity and other interested audi* 
ences a sampling of empirical outcome studies conducted ^ 
since the enactment of the Vocational Education Miendments 
of 1968 f which mandated evaluation of vocational prograimSt 
To a limited extent, it is a response to a call for a de- 
tailed review of the methodology and substance of evalua- 
tion studies*. ^ 

Vocational education outcomes are broadly defined as 
the consequences of vocational programs. These conse- 
quences can be attributed to (a), a student^s participation 
in a vocational program and/or (b) . the existence of ongo-- 
ing vocational programs in the community* Outcomes may af- 
fect an individual student, society as a whole, or some 
segment of society such as taxpayers, employers, or a spe- 
cific community* Vocational programs may be at the secon- 
dary or postsecondary level; they may include programs for 
adults and out-of-school youth. All results, consequences, 
impacts, or effects of vocational education are considered 
outcomes, whether positive or negative, intended or unin- 
tended, short-term or long-term, economic or noneconomic* 

Two types of documents are included in this bib- 
liography. The main body of the bibliography lists em- 
pirical outcome studies^ that is, studies that report and 
analyze data concerning the outcomes of vocational pro- 
grams. The studies utilize data systematically collected 
and analyzed in a way that can be verified and/or repli- 
cated by other researchers. The appendix includes a selec- 
tion of literature pertinent to the study of vocational 
education outcomes* 



The proposal for such a review was made by Professor J* 
Robert Warmbrod in his presidential address at the Annual 
Meeting of the American Vocational Education Research Asso- 
ciation, Houston, Texas, December 6, 197 6, "Evaluation Re- 
search in Vocational Education," reprinted in the Associa- 
tion's nav^^Bl^ator Beacon, Vol, S, No* 1 (January 1977) 8. 
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The axt^hsive search for empirical outcome studies as 
defined above yielded a body of literature cloeely related 
to the study of vocational education outcomes i data-base 
reports I review and synthesie papers i evaluation methodo^ 
logy revaluations focusing on processes or other non- 
outQome variables; and other diverse documents. Although 
this literature does not fit the narrow topic of "voca-- 
tional education outcome studies," it is fundamental to 
■ a contextual understanding of those studies* Further ^ 
many of the documents, such as "the GAO Report" and "Pro- 
ject Baseline," are often informally cited in the litera- 
ture* Therefore, an appendix with the broader topic of 
"literature related to the study of vocational education 
outcpmes" is included, with full citations and descriptive 
annotations of these important documents. 

This selected bibliography may be useful to a variety 
of audiences — from experienced vocational education spe^ 
cialists to employment and training planners and interested 
laypersons. The annotations for both the outcoma 
studies and the key related literature were written with 
the assumption that some readers with little background in 
vocational education or educational research may have real 
and pressing needs for pertinent information that can in- 
fluence vocational education policy. This document, then, 
will provide a concise sourcebook on vocational education 
outcomes reaearch for policy analysts at all levels of 
government, for academic researchers, for staff members of 
state departments of eduoation and local school systems, 
and for practitioners in other fields of education and 
training. 



A-4 



I. INTRODUCTION TO THE EMPIRICAL 0UTC0MJ3 STUDIES 

This bibliography contains descriptive and evaluative 
annotations of a selection of empirical studies assessing 
the outcomes of vocational education. It was coitipiled pri- 
marily to provide the vocational education community and 
other interested audiences a sampling of substantive find- ^ 
ings and methodological issues and to give suggestions £or_ 
Surther research. Vocational education outcomes are broadly 
defined as the full range of consequences that can be attrib- 
uted to vocational programs, and empirical studies are de- 
fined as those studies in which data are oolleGted and ana- 
Ivzed in a way that can be verified and/or replicated by othei 
researchers. Thus, the bibliography includes such documents 
as reports of follow-up studies of the graduates of particu- 
lar vocational education programs, annual evaluation reports 
prepared by state vocational education advisory councils, 
and doctoral dissertations addressing specific vocational ^ 
education outcomes. Excluded from the bibliography proper 
but included in the appendix are general data-base reports j 
review and synthesis papers; evaluation methodology papers? 
evaluation studies focusing on student characteristics, con-, 
text, resources, goals, or processes of vocational education r 
and other publications that are related to vocational edu- 
cation evaluation. 

The studies annotated in the bibliography were selected 
through a two-stage process of identification and review. 
In the first stage, a variety of sources was consulted in 
a search for documents addressing topics under the broad 
heading of vocational education evaluation. This encompass- 
ing identifier was used in the initial stage since the more 
narrow term "vocational education outcome studies is in- 
frequently used and does not even elicit iiranediate recogni- 
tion from many vocational education researchers. During 
this first-stage search, National Center staff and other 
vocational education professionals were consulted, two com- 
puter searches were initiated, 1 and bibliographies of al- 
ready collected documents were screened. 



search of the Educational Resources Information Cen- 
ter (ERIC) system was conducted and references were retrieved 
and scanned for relevance to the project. Among the descrip- 
tors used, singly and in combination, were vocational educa- 
tion, vocational follow-up, measurement, and program evalu- 
ation A search of the Comprehensive Diss ertation Index 
was "also conducted — (footnote continued on the next page) 
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In the second stage ^ the identified documents were re- 
viewed* Those not reporting empirical studies of the con=- 
sequences of vocational education were excluded from the 
bibliography proper* Some of these documents such as de^ 
scriptiva reports of vocational education, evaluation hand-^ 
books^ literature reviews , and papers providing only anec^ 
dotal evidence concerning educational outcomes are listed 
and annotated in the appendix of key related literature* 
The remaining documents were judged to be bona fide empiri- 
cal evaluation studies. Those retained for inclusion in the 
bibliography were selected to 

1* reflect the broad scope of vocational 

education outcomes research 
2* utilize primary sources insofar as possible 
3* illustrate evaluation research and analysis 

methods as well as findings 
4, identify research constraints and related 

problems - ^ " 

Using this identification and review process, thirty^ 
one studies were selected for the bibliography, Bibliog^ 
raphies of those documents were reviewed for other studies 
and National Center staff and outside consultants were asked 
to identify other studies that should be included* Although 
several other studies exist, these sources indicated that 
they were not so different from those already identified as 
to constitute important additions. Thus, this listing does 
not exhaust the available literature, nor is it a random 
sample of the qualitative variation in research activity* 
Basically, however, it is indicative of both the substan-- 
tive and methodological range of contemporary evaluations 
of vocational education outcomes. 

Each annotation begins with discussion of the research 
questions and hypotheses; the study design, data sample, 
and time frame; the variables and statistical analyses (if 
any); and the results and conclusions. It concludes with 
an evaluative segment addressing some of the salient meth-- 
odologleal and substantive issues inherent in each study. 



footnote continued =^ and references retrieved and 
scanned* Among the descriptors used were vocation (al) , 
occupation ( al) , postsecondary , community college^ secondary, 
adult, dropout, outcome, impact, followup, evaluation, ef- 
fective (ness) , benefit, cost, minority, women, Indian^ handi- 
cap (ped) , disadvantaged, bilingual, migrant^ multicultural^ 
blind* Combinations of these and other descriptors were 
used as well* In both searches, most of the documents Iden-- 
tified were not reports of empirical outcorie studies, 
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Although applied researchers^ includinGr evaluators/ must 
bailee it^thodological concerns with conGerns for factors 
aiffeatlng the utility of results (such as timeliness and 
politiaal viability) / the latter issues are more difficult 
to assess through reading an evaluation report. The evalu-- 
ative Gomments^ thus ^ are intended not to serve as meta- 
evaluations^ but to alert the reader to factors influencing 
the credibility of the results. Therefore, questions^ are . 
raised concerning such issues as appropriateness of design, 
generali^ability of findings, consideration of relevarit viiri- 
ables, and clarity of reporting* Notable features and 
strengths of the study may be highlighted. The substance 
of the annotations is further discussed in the "Technical 
Notes" following this section. 

ED numbars, where provided in the citations^ refer to 
the accession nundser prefix sequentially assigned to docu*- - 
ments as they are processed into the Educational Resources 
Information Center (ERIC) system and published in the monthly 
abstract journal^ Resources in Education (RIE, published by 
the National Institute of Education), The docmnents, ex- 
cept as noted in RIE, are available fron the ERIC Dociment 
Reproduction Service (EDRS) / in either microfiche or hard 
(paper) copy. The address isi P* 0, Box 190^ Arlington/ 
Virginia 22210, AlsO/ many university libraries have col^ 
lections of ERIC microfiche. When no ED number is given 
for a particular document^ it presumably has not been 
entered in the ERIC system. 



II, TECHNICAL NOTES ON ANNOTATING 
THE EMPIRICAL OUTCOME STUDIES 



Evaluation is a term that has several meanings/ depend' 
ing on the orientation of the user. A standard educational 
evaluation text^ lists five different usages of the term. 



For a discussion of standards and guidelines for eval-- 
uating evaluations/ see Daniel L, Stuff lebeam. "Meta^ 
evaluation . " Occasional Paper Series (No, 3) . Kalamazoo: 
The Evaluation Center/ Western Michigan University, 1974, 

^Blaine R, Worthen and James R, Sanders. Educational 
Evaluation: Theory and Practice , ((Worthing ton / Ohioi 
Charles A* Jones Publishing Co./ 1973) ^. 20. 
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For this bibliography^ we did not choose any one meaning, 
but rather annotated documents that are characteriEed as 
evaluation reports by their authors or that are used as eval- 
uation reports by some audience (for a diseussion of diverse 
definitions of avaluation^ see Attachment "C") . However, we 
maintain that regardless of definition, the process of 
evaluation is a process of inquiry. Although evaluation as 
a generia term may differ from research, 4 whan evaluators set 
out to collect data (that is, to inquire) , they are not 
exempt from the canons of research, from the scientific 
tenets of "the conduct of inquiry."^ Although evaluators 
must balance concerns for methodological issues such as va-* 
lidity with concern for practical issues such as timeliness, 
it is our contention that a timely evaluation report of 
questionable validity is a more serious error than an un- 
timely but valid report. The latter report may be of use 
to an" audience other than that for which it was intended, 
wheraas the former is of little use to anyone. Thus, al-^ 
though empirical research is not the only legitimate evalu- 
ation strategy, and a case may be made for a multiplicity 
of evaluation strategies (including the case study approach, 
for instance) , the evaluative comments are made from an 
empirical research perspective. 



Descriptive and Evaluative Annotations 

A checklist of important components of an evaluation 
report was developed® and used to organize each descriptive 
annotation (see Table I) * As alluded to in the above dis- 
cussion, the outline is similar to one that would charac- 
terise any other scientific research report. 



For a discussion of this issue, see Worthen and 
Sanders. Educational Evaluation . 

^A short bibliography of research design texts and 
evaluation report checklists will conclude these technical 
notes. See, for instance, Abraham Kaplan, The Conduct of 
Inquiry I Methodology for Behavioral Science . (Scran ton, 
Penn. : Chandler, 196 4) , ~~ 

^These components wmrm not drawn from any one source; 
however, discussions of them can be found in several re- 
search design texts such as those listed in the reference 
section concluding these notes* Terminology and relative 
emphasis vary among authors (reflecting diversity in usage 
by researchers) , but there is little disagreement about the 
basic components. 
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The evaluative coiranents are more judgmental than the 
desariptive annotations^ both in their substance and in the 
selection of issues on which to express professional judg- 
ment.* Although the comments address the methodological 
issues surrounding the components of the evaluation report 
oheoklist (Table I), space limitations prevent detailed 
Gritiques of every issue The evaluative comments focus 
on selected aspects of credibility and policy utility i from 
the information given in the report, (ij to what extent are 
the conclusions warranted , (2) are the conclusions relevant 
to the stated purpose of the study ^ and (3) can the conclu- 
sions be generalised in such a way that they contribute to 
a body of knowledge about the effects of vocational educa- 
tion? Suggestions for improving further research are madef 
and the completeness and clarity of the reporting are ad- 
dressed as well. 

To provide a basis for the assessment of credibility 
and utility/ oertain information must be presented in the 
report and that information must indicate that oertain meth- 
odological practices were followed* The evaluative conunents 
focus on some salient issues. For example , what aspects of 
the report contribute to the reader's credence in the con- 
clusions? What was done especially well, and what was lack-- 
ing? It cannot be oyerly stressed that we do not address 
all the issues exhaustively. 

Following is a brief discussion of the issues that are 
emphasised. It should be understood that although an ideal 
study would be methodologically perfect/ situational con- 
straints and imperatives (lack of adequate time^ money # ex- 
pertise, . * .) can prevent the goal from being reached; 
these constraints and imperatives are not always discussed 
in an evaluation report. The negative comments therefore 
are not meant as accusations but are intended to alert the 
reader to factors attenuating the credibility of the 
conclusions. 



A thorough critique of an evaluation report could 
easily take five to ten pages; and some are even longer* 
See/ for instance / the critiques published in The Journal 
of Vocational Education Research ^ Vol. 1, No, 1, Winter 
1976. 
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Table I 

CHECKLIST OF COMPONENTS OF AN EVALUATION REPORT 
Statement of purpose/ hypotheses, questions 



Contextual factors 



Population , sample 



Design^ data, procedures, instruments 



Data treatment and analysis 



Reaults 



Cbnclus ions / implications , recoimnendations 

Sponsor of study , professional identity 
of evaluator 



Completeness and clarity of Reports 

It is important that sufficient detail be included in 
the report to allow readers to mentally reconstruct the pro 
ceduras followed^ the instruments used, and the groups stud 
ied. The readers need this information to judge the appro- 
priateness of the methodology and to determine how similar 
the programs and students studied are to those they are com 
paring. Thus, not only must all the requisite information 
be given, it must be provided without ambiguity. Terms 
such as "program compietGrs" and "early leavers" must be de 
fined, differences between such groups as "vocational stu-- 
dents" and "non-vocational students" must be eKplained, and 
the exact data collection and analysis procedures must be 
described . 

Multiple reports are sometimes written, each with in-- 
formation important to a specific audience. In such cases ^ 
each version may not contain all the information discussed 
here* However, when multiple reports have been written, 
each report should make reference to the other reports, in- 
dicating the audiences to which they are addressed and the 
scope of the included information. 
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When inquiry is conducted for eKploratory purposes ^ 
detail sufficient to enable the readers to grasp the general 
idea of what was done is adequate. However, when research 
is done to verify an hypothesis^ the amount of detail must 
be such that researchers who read the report can replicate 
the procedures if they wish* (The topic of verification is 
covered more fully in the neKt section.) The -acquisition of 
a body of knowledge is dependent on replication cpf research, 
although statistical analyses can indicate the likelihood 
that an observed phenomenon is a function of chance, ^ 
peated replications of procedures yielding the same findings 
are necessary to provide further assurance that they were 
not chance occurrences * 



Statement of Purpose^ Hypotheses , Questions 

When engaging in inquiry, one must distinguish between 
the "logic of discovery" and the "logic of justification." 
The logic of discovery' applies to the formulation of hypoth^ . 
eses. There are several methods of formulating hypotheses : . 
for exMiple, they may be based on theory, they may be ground- 
ed in observation, or they may be strictly hunches,^ There 
are no formal rules that eKhaust the possible procedures for - 
discovery of hypotheses, but in both scientific and everyday 
discourse, a number of formal procedures for confirming or 
verifying hypotheses can be identified. ^ These procedures 
fall under the rubric of the logic off justification? some 
of them will be discussed in the following sections. Because 
the suitability of methodological procedures depends on the : 
purpose of the study (discovery or justification) , it is 
important that an evaluation report include a purpose state- 
ment* 

The specific objectives, which should be stated as evalu- 
ation hypotheses or questionB, as well as the overall purpose i 
will have implications for the choice of methodology. Again, 
in order for the reader to judge the suitability of the metboc 
ology, the specific objectives of the evaluation endeavor 
must be stated j The reader is cautioned to review the method' 
ology used in every study to see if it is appropriate and ade- 
quate to answer the questions posed. 



®Some philosophers of science suggest that an hypothesis 
derived from a particular theory is more acceptable than one 
derived from no particular substantive context. See, for ex^ 
ample, Karl R. Popper. The Logic of Sci entific Discovery, 
Rev. ed. (Londoni Hutchinson ana CO., 1968), Chapter 30, 

^See, for eKample, Kaplan. The Conduct of Inquiry i 
Methodology for Behavioral Science. 
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ConteKtual FaQtors 



Any eOntextual factors that may affect the interpreta- 
tion o£ the results should be disdussed. The time frame is 
an important example i When was the study conducted? What 
time elapsed between the occurrence and the study of it? 
Whan were the data analyzed and reported? These time fac- 
tors are important because, for aKamplei (1) Some event may 
have occurred during the study affecting its results/ even 
though not directly related to vocational education* (2) 
One may question the accuracy with which respondents recall 
details of their first jobS/ if questioned ten years later. 
(3) The accuracy of the researcher's recollections is ques-- 
tionable when data are analyzed two or three years after 
collection* Other contextual factors include geography and 
events that are not necessarily indicated by time-frame in= 
formation (events with far-reaching consequences are gener-^ 
ally known by their dates ^ but more local events are usually 
recalled from a different standpoint) . 



Population and Sample 

The study's population is the universe of people, ob- 
jects^ or events in which the evaluator is interested. Sta- 
tistically significant results of a scientific study can be 
generalized to the designated population if appropriate 
sampling techniques are used (treated further in the Data 
Analysis section) . Thus^ the population must be specified . 

In some cases, researchers are able to cQllect data 
from or concerning the entire population. In most cases 
this is not feasible, however, so the researcher selects a 
sample from the population. Appropriate sampling techniques 
must be used to ensure that results from the sample are rep- 
resentative of results that would be obtained if the whole 
population were studied. The sampling procedure used, there- 
fore/ must be specified. Sampling teahniques and issues are 
described in various survey research texts as well as. in 
some more general research texts. 

When a sample is chosen but some data are missing be^ 
cause not everyone (a) was available for an interview, (b) 
returned the questionnaire, or (c) answered all queations, 
the subset actually used in the study is sometimes referred 
to as the study sample. What constitutes the study sample 
and the bases on which it differs from the original sam-- 
pie should be specified. 
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It is important that full information be reported con- 
cerning the population, the sanaple, and the study sample. 
Tarns must be defined, procedureB must be described (how the = 
sMipla was chosen, what rules were used to determine which 
data were unusable . , and numbers must be provided 

(sige of the population, size of the sample, size of the 
study sample) . For example, if it is merely stated that the 
response rate was 64%/ it is not clear whether the study f 
sample was 64% of the population or of the original sample* 
Conclusions reached on the basis of a large percentage of 
returns from a small but carefully selected smple may be 
more valid than those based on a small percentage of returns 
from a large population even if the population and the study 
sample sizes are equal in both cases. 

Bias is an important issue related to sampling* When 
the original sample and/or study sample are not representa- 
tive of the population, they are said to be biased. Biased 
samples can be purposefully selected, as when a teacher con-- 
siders only the best students and then reports that most, of 
the students got jobs; or they can occur beyond the control 
of the researcher, as when students without jobs fail to rec- 
tum a questionnaire asking about their employment status* 
Not only should researchers choose sampling and data-gather-^ 
ing techniques to minimize bias, but they should report rel* 
evant characteristics of the study sample in comparison with 
characteristics of the population in an effort to determine 
the existence of bias. For example, in a study where sex 
may make a difference, the ratio of males to females in the 
study sample should be compared with the ratio of males to 
fCTiaies in the population. Statistical techniques can be 
used to overcome problems of bias in some situations.-^ 

The size of the study sample is one of the factors con- 
tributing to the power of a statistical test. The study 
sample must be large enough to ensure that if there are true 
effects, they can be statistically demonstrated* This is 
somewhat akin to needing a magnifying instrument- powerful 
enough to make observable something that exists but cannot 
be seen with the naked eye. 



Design, Data, Procedures, instruments 

The choice of research design, procedures, and data- 
collection instruments is influenced by such factors as 



lOSee, for example, William G. Cochran* Sapling Tech -- 
niques* (New Yorks John Wiley and Sons, 1977) . 
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purpose, available resources, and political constraints; 
trade-offs are inevitable. Although the readers of evalua- 
tion reports are generally not in positions to judge the 
choice of such specifics, they should be able to follow the 
avaluator's logic in moving from purpose to design* 

The basic design issue is validity. Does the chosen 
design allow warranted conclusions to be drawn about the 
evaluation questions or hypotheses? One^must ^estwn ^ 
whether somiting other than the treatment Un thxs conteKt, 
vocational education or some aspect thereof) can be respon- 
sible for the observed effect. There are many threats to 
validity and several research designs developed to attenuate 
them. A brief summary of a few of the common research de^ 
signs, adapted from a classic text, is presented in Table II. 



Table II 
SOME COMMON RESEARCH DESIGNS 

1, The One-group Pretest = Posttest Designi One group 
receives a pretest, a treatment/ and a posttest* 
There is no control over the possibility that factors 
other than the treatment were responsible for any 
changes that occurred. (If this design is used/ a 
correlated t-=test is an appropriate statistic for data 
analysis . ) 

2. The Static-group Comparisoni Two intact groups are 
chosen* Only one receives a treatment; both receive 

a posttest* There is no adequate assurance that both 
groups were essentially the same before the one group 
received the treatment* (If this design is used, a 
t-test is an appropriate data analysis technique*) 

3. The Time^series Experiment t A series of measurements 
is taken on a group, a treatment is given, and the 
series is continued* Something other than the treat- 
ment may have caused any observed change in pattern, 
(Many factors must be considered in choosing the ap- 
propriate statistical analysis*) 

4, The Nonequivalent Control Group Designs Two intact 
groups receive a pretest/ one group receives a treat- 
ment, and both groups receive a posttest. Control 
of some of the threats to validity depends on the 
situation* 

Source I Adapted from Donald T. Campbell and Julian S, 
Stanley , Experimental and Quasi -EKperimental 
Designs for Research* Chicago ^ Rand McNairy/ 
1963. " _ _ 
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In addition to deciding on a general research desigrir 
the evaluator must make several specific decisions relative 
to data^ procedures^ and instrumentation i (1) What is the 
nature of the data to be collected? (2) From whom should 
data be collected? (3) When should data be collected? and 
(4) HOW should data be collected? The evaluation report 
must indicate the chosen alternatives and their rationale 
in sufficient detail that the reader may judge whether the 
data, procedures ^ and instrumentation are appropriate and 
adequate to the purpose of the evaluation. These four issues 
are interrelated and complex; only a few of the important 
factors will be discusBed here. The research design texts 
listed at the conclusion of these notes provide much greater 
detail. In addition, the book by Stuff lebeam and others . 
(1971, p. 22) discusses the limitations of research paradigms 
for evaluation studies. 

The general class of data necessary to answer the pur- 
pose of an evaluation will be mpre or less obvious depending 
on the specificity of the purpose statement^ and it is the 
evaluators who must convince their audiences that they have 
chosen appropriate outcomes to study and appropriate data to 
assess those outcomes..ll Not only must an outcome---f or in- 
stance, earnings— -be specified, but it must also be opera- 
tionalized, that is, it must be defined in a way that clearly, 
indicates how it is to be measured. Further, the evaluator 
must justify the operationalization—for instance, on what 
basis was a choice made from several possible choices, such 
as weekly earnings, annual earnings, and average earnings* 

The general question of data sources raises two more 
specific questions: (1) Should the data be collected directly 
from mOTibers of the population or from other sources? and 
(2) What comparisons should be made? For example, informa- 
tion about the job duties of a population of employees could 
be sought from a sample of the employees or from their em- ^ 
ployers,' Tradeoffs among such factors as accuracy, feasibil^ 
ity, and cost are sometimes necessary in choosing a data 
source. The evaluation report should provide a rationale 
for the choices made. 

If conclusions about the effectiveness of vocational 

education are to be drawn, some comparisons must be made. 

The variables being studied indicate what should be compared, 

e.g., earnings, but for purposes of evaluation, these 



llgee Attachments "B" and "C" for further discussion of 
this point. 
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variables must be compared against some standards* The 
standards may be relative^ e.g*/ one group of graduates com- 
pared to another, or they may be absolute, e.g,, an employee's 
performance compared to a stated level of satisf aotorinees . 

The choice of a standard or a comnarison group is rela- 
tively easy when there is a theoretical reason to suspect 
that the performance of a group should be different after 
its participation in a vocational education program or dif- 
ferent from that of a group that did not participate* Ideally, 
the theory would even specify the nature of the difference * 
When evaluators make comparisons, they should specify the ra-- 
tionale for their choice of standards. The reader then can 
judge whether appropriate comparisons have been made* When 
standards are not stated it should be indicated that it is 
left to the readers to compare the descriptive data presented 
with whatever standards they prefer. 

How the data are to be collected is another issue that 
involves many tradeoffs • The best technique is often the 
most expensive (see Table III for some data-collection meth= 
ods) . Once a method, such as personal interviews or mailed 
questionnaires, has been chosen, an instrument must be select-- 
ed as well. An instrument may be developed for the research 
project or one already available may be chosen. In either 
case, questions of validity and reliability should be ad- 
dressed as well as conditions of administering the instrument. 



Table HI 
SOME DATA COLLECTION METHODS 


METHOD 


STRENGTHS 


WEAKNESSES 


Machanical Devices 
( a . g . 1 videotape ) 


Elimination of human 

errors 
Reliability 


High cost 
Complexity 


Observation 


Natural settings 

available 
Subtleties perceived 

by trained observers 


Artif ical situation 
created by observer *s 
presence 
Variation in reliability 
Meaningful factors pos- 
sibly overlooked by 
observer 



See Eugene J. Webb and others. Unobtrusive Measures i 
Nonreactive Research in Social Sciences , ( Chicago t Rand ~ 
McNally, 1966) for a discussion of comparisons, pp. S^-lO * 
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Table III (Continued) 
SOME DATA COLLECTION METHODS 


METHOD 


STRENGTHS 


WEAKNESSES 


Questionnaires 


Honest# fostered by 

anonymity 
Low cost 


Possibility of low per- 
centage of returns 

Respondents^ lack of 
understanding 

Superficial responses 
easy 

Return biased by nature 
of the questions 


Interviews 


Provision for flexi^ 

bility 
Provision for response 

in depth 


High cost 

Skilled interviewers re- 
quired 

Difficulty of suiranarizing 
Many biases possible 


Rating Scales and 
Checklists 


Low cost 

Ease of completion 
Objective scoring 
Standardized response 


Not all alternatives en- 
compassed 
May tell more about respon* 

dent than about topio undei 
gonsidaratlon 


S el f= Reports 


Economy 

Inclusion of emotions- 
laden material 


Difficulty of analysis 
High degree of subjectivity 
May be threatening 


Tests 


Assessment of reli- 
ability 
Analysis of items 
Ease of scoring 


Validity of questions 
doubtful 

Other methods often eK- 
eluded 

Triviaixty or questions 

Other alternatives not 
covered 

Precision of format mask- 
ing poor items 


Performance 
Smples 


Direct measure of 
ability and mastery 


Difficulty of adminis-- 

taring 
High cost 

Problem of reliability 
Artifical situation 


Unobtrusive 
Measuras (e.g. t 
"wear on car- 
pet") 

Source: Adapted f 
Education 


Nonreactive 

Not consciously biased 
Availability and 
measur ability 

rora material presented in ^ 
al Evaluation and from oth 


Quastioni of ethics 
Questions of validity 

terthan and Sanders 
ar sources , 
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It is irnportant that an instrument he both valid ^ that 
is measure what it is said to measure, and reliable, that 
is consistent in its maasurement , Neither validity nor re- 
liability is inherent in an instrument; hoth depend on factors 
of the situation in which the instrument is used. Still, if 
an instrument has been demonstrated to he valid and reliable 
in a given situation and it is to be used in a similar situ- 
ation^ assertions of probable validity and reliability are 
reasonable . 

There are several kinds of validity and corresponding 
ways to assess^^ them but for present purposes an example 
will suffice. To assess the capability of Test A to measure 
the ability of individuals to perform Task X, the test can 
be given to individuals known to have different levels of X 
ability. If the variance in scores on Test A corresponds with 
the known variance in ability to perform K, the test is said 
to have concurrent validity. 

Although there are several ways to assess reliability/ 
it is a unitary concept. If a measuring device consistently 
yields the same measurement when applied to a phenomenon, it 
is said to be reliable. Thus, a thermometer that always reads 
lOO^C whan placed in boiling water is a reliable instrument, 
whereas one that varies, either randomly or with some other 
variable, is not. Although the thermometer is just as reli-- 
able if it always reads 90^0 when placed in boiling water, it 
cannot validly measure water temperature. The reliability of 
tests of knowledge is assessed by comparing scores on alterna- 
tive forms of a test, by comparing scores on odd items with 
scores on even items (assuming that overall, all items test 
the same pool of knowledge), or by comparing scores on two 
different administrations of the same test. 

Reliability and validity are difficult to distinguish 
and to assess with certain measures of the hypothesised out-^ 
comes of vocational education. Still, it is important to 
assess them insofar as possible. A questionnaire should not 
be assumed to elicit required information, from the point of 
view of the researcher, without some "reality-check*" Nor 
should responses to an interview be assumed to be factual , 



Predictive validity, content validity, construct val- 
idity, and face validity are among the several types of val- 
idity that can be assessed. See J. P. Guilford. Fundarnental 
Statistics in P sycholog y and Education (New York: McGraw-- ^ 
Hill, 1965) . 
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Questions may be misinterpreted in some cases, and in others 
a raspondent may not have accurate information or^ having it, 
may not want to give it. Questionnaire items may be mlsurider- 
stood by all individuals in a sample or by some specific sat 
of Inclividuals . 

Standard techniques found in research texts for assess- 
ing validity and reliability often cannot be applied to mea- 
sures of vocational eduGation outcomes because they are de^ 
signed for such purposes as multiple-choice tests 
of knowledge, attitude scales , or performance tests. Still, 
it is. necessary to determine if the techniques used to gather 
vocational education outcomes data are valid and reliable, 
A general method that can be used in many instances is pllot^ 
testing. To pilot^test ' an instrument or other data--gathering 
technique/ evaluators select a small representative sample 
from the population of interest and collect data from that 
sample. They then attempt to verify the information received. 
One procadure is to discuss the data collection device vrith 
the members of the sample to determine their understancaiiig of 
what was being asked. Another is to elicit the same informa^ 
tion frcOT another source* For example, if parents are gues- 
tioned about employment of their children, their information 
can be verified by questioning the children themselves a nfl 
their stated employers. If results of a pilot test indicate 
that the measuring devices accurately measure the content of 
interest to the evaluator and do so consistently, it is rea- 
sonable to assume that the devices are valid and reliable 
for other samples from that population. Another method that 
can be used to support claims of validity is the use of a 
panel of sub ject-^matter experts* The panel would judge 
whether the instrument, for example, a questionnaire or intsr-^ 
view schedule, covered the domain of interest, 

Another factor related to data collection should be dls^ 
cussed in an evaluation reporti the conditions of instrument 
administration. These conditions may impact on reliability 
and/or validity. For example, little credence can be placed 
in interview data obtained from untrained interviewers ox 
from data collected under uncomfortable physical conditions 
(for example ^ questionnaires completed in a cold room) . 



Data Analysis 

For some purposes, simple presentation of data may be 
sufficient. Even in these cases, though, care should be 
taken to guard against misleading tables. When statistical 
analyses are used, even greater care must be taken. Many 
assumptions and conditions underlie most statistical techniques 
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and although many can be violated without detracting from the 
usefulness of the statistic, some cannot be, 14 It is partic^ 
ularly important to meet assumptions when paraineters such as 
unemployment rates are being estimated, whereas when patterns 
of outcomes such as job satisfaction are being explored, vio- 
lations of assumptions may not be so critical. 



Results 

Interpretation of statistical findings must be done with 
care, 15 Statistical significance indicates that an observed 
effect (for eKample, higher salaries for vocational graduates) 
is not likely a function of chance variation in the selected 
sample but further statistical tests can sometimes be done to 
indicate how much of the variation is a function of the treat^ 
ment in question (for eKample^ vocational education) . With 
correlations, for eKample , when the correlation coefficient 
is squared and then multiplied by 100, the resultant percentage 
Indicates the variance explained. With large sample sizes, 
statistical significance is often found even though the ex- 
plained variance is very low; when this occurs, it indicates 
that the results have little practical significance, A care-- 
ful reading of the report is needed to determine whether the 
stated results are an outgrowth of the reported data. 

Conclusions , Implic ations , Recoimiendatlons 

Some evaluation reports may simply present data and sum- 
mary results, leaving the policy maker to dra^ cDnclusions. 
When conclusions are drawn, implications stated, and/or rec-- 
oimnendations made in a research report, they should be based 
on the data* 



For example, assumptions that the population is homo- 
geneous and that the variables under study are normally dis- 
tributed in the population. These assumptions are treated 
in statistical texts, 

1 ^ 

See Darrell Huff. How to Lie with Statistics , 
(New YorKi W. M, Norton and Company, Inc, 1954) . 
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Sponsor of Study. Profes sional Id entity of Kvaluator 

The Identity of the sponsor and of the evaluator can 
affect the credibility of an evaluation report, both positive- 
ly and negatively. Some sponiors may be knovin for requiring 
quality work and others for preferring results that support 
their positions. Similarly, some evaluators are recognized 
for accurate, unbiased Inquiry while others become so in- 
volved in the subject of their inquiry that their results are 
suspect. ^® 



••■^See Ernest Nagel, "The Value-Oriented Bias of Social 
Inquiry," in May Broadbeck (ed.)' Readings m the Philosophy 
of the Social Sciences. (New York: The Ilacmillan Co . , 1968), 



50 

60 



RFFP^.ENCES POR THE TECHNia^J. NOTES 



Adams, Kay^ Angona and Walker, Jerry P* Improvl iig^ the AccQunt*- 
ablllty of Career Educatioii Progr ains i Evaluation Gulde--> 
lines and Checlcl ists. ~ Columbus z ~f he Center for Voca- 
tidnal Education, The Ohio State University^ 1977* 

Campbell, Donald T. and Stanley, Julian C* Experiingntal and 

Quasi-EKperiment al Peslgns for Research « Chicago i Rand^ 
McNa 1 ly , 196 3". 

Cochran, William G, Samp ling Techniques , New YorJc^ John 
Wiley and Sons, 1977, 

Converse, Jean M. and Schuman, Howard. Conversations at Random 
Survey Research as Inte:cviewers See It . New Iforki John 
Wiley and Sons, 1974. 

Edwards, Allen Probability and Statlstica . Ne%^ YorKi Holt 

Rinehart, and Winston, Inc * ^ 1971. 

Parquhar, William W* and Krumbolti, John D. "A Checlclist for 
Evaluating EKperimental Research in Psychology and Educa- 
tion," Journal o£ Educational Research , Vol, 52, No, 9, 
May 1955": ^ _ 

Gorden^ Raymond L. Intervie wing^ Strategy, TeahnigueS/ and 
Tactics . Homewood, Illinois I The Dorsey Press, 1969, 

Gray, Kanney E., Abram, Marie, and HcKinney, Ployd L, Voca- 
tional Education Measures i Inatru nients to Survey Foriner 
Students and~f heir Employers . Columbus i The National 
Center fo^ "Research in Vocational Education, The Ohio 
State University, 1978. 

Guilford, J* P. Fundamental Statistics in Psychology a nd Edu - 
cation . New YorSTi McGraw-Hill , 1978~ 

Hopkins^ Kenneth D. and Glass, Gene V* Basic St atiatlcs for 
the Beh avioral Sciences. Englewood Cliffs, N.J. t Pren- 
tice Hall, 1978." 

Huff, Darrell, How to Lie with Statistics > New York i W, W. 
Norton and Company, Inc * , 1954 * 

Kaplan, Abraham. The Conduc t of Inquiry^ Meth odolpgy for 
Behavioral Science . Scranton, Penn,i Chandler^ 1964. 

Ker linger, Fred N. Foundations of Behavi oral Research* New 
Yorki Holt, Rinehart, and" Wins to~n, Inc., 1973. 




A-24 



Nagel^ Ernest* '*The Value-Oriented Bias of Social Inquiry i" 
in Readiiigs in the Philosophy of the SoQial Sciences ^ 
edited by May Broadbeck. New Yorks The Hacmillan Co*^ = 
1968. 

Payrie^ Stanley. The Art of Asking Questions. Princeton i 
Princeton university Preis^ 1951, 

Popper^ Karl R* The Logic of S cientlf ic Diecovery . Rev, ed. 
London i Hutchinson and Company ^ 1968. 

Sanders, Jamee R, and Nalziger, Dean A Basis for Determine 

inq the Adequ ac y of Evaluation D eslgni. Kalamaioos 
The Evaluation Center, College of Education, Western 
Michigan University^ 1977* 

SaK^ Gilbert. Empiric al Foundations of Educational Researah . 
New Yorki Prentice' Hall , 1968 . 

Siegai, Sidney. Nonpara nietric Statistics. New Yorki McGraw- 
Hill, 1956 , 

Stuf flebeam, Daniel L. "Meta-evaluation, " Occasional Paper 
Series (No, 3). Kalamazoo i The Evaluation Center, 
Western Michigan University, 1974. 

Stuff lebeam, Daniel L., FQley, Walter J., Gephart, William J., 
Cuba, Egon G* , Haimnand^ Robert L, , Merriman, Howard 0./ 
and Provus, Malcolin M. Educat ional Evaluation and Deci - 
sion Making . Itasca, Illinois: Peacock, 1971. 

Sudman, Seymour and Bradburn, Norman M. Response Effects in 
Surveys I A Re vlev^ and Synt heiis , Chicago % Aldine ^ 
Pubrishrn~g Co-', 19 74. 

Tuckman, Bruce W. Conducting Education al Research . New York 
Harcourt, Brace i Jovanovlch , 197 8 • 

Measuring Educational Qutcomes. New York: Harcour 
Brace , Jovanovich, 1975, 

Van Dalen, Deobold B. Understanding E ducational Research , 
New York I McGraw-Hill Co/, 1973 .. 

Webb, Eugene J., Campbell, Donald T,, Schwartz, Richard D., 
and Sechrest, Lee. Unobtrusi ve Me asures i Non reactive 
Research in Social Sciences, Chicago i Rand McNally, 196 

Worthen, Blaine R. and Sanders ^ James R, Educational Evalua - 
tion i Theory an c[ P ractice . Worthington , Ohio : Charles 
A* Jones PubTishing Co., "1973. 



52 



A«-25 



III, EMPIRICAli OU'PCOMli: H'l'lIDIKK 

A. List of Titles^ 

' Page 



The Adequacy of Vocational and Technical Education , 5g 
" South Carolina Advisory Council on vocational 
and Technical Education, 1976. 

Animal Hea lth T echnicla na! A Survey o f Program 59"^ 
" Graduates and of Veterinarians . "Richard B, 
Barsaleau and Henry R, Walters, 1977. 

An Aese asment of Benefits Derive d from Membership (5] 
in a Vo cational Studen t Organisa tion in the 
Vocational", TeahnXcB.1 , and A dult Education 
System . Dennis R, Collinsr 1977, 

Benefit-Cost Comparison of Vocat ional Education G3 
Programs ." Marshall AT. Harris ^ 1972 . 

CQmparative A nalysis of Pos tseco ndary Occupatio nal 65 
and Educational" Outcomes for the High School 
Class o f 1"972, P . Re"id Creech and "others / 
1977/ 

A Comparative Study of th e Oocupational Achievement £8 
or Voca tional and Non-Vocational High School 
Graduat e s in t he Sta t"e~o? Ar kansas. Dean CT 
Andrews and Lawrence H. Roberts, 1974. 

A Comparative Study of the Occupational Achieve - 7G 
ment of Vocat ional and Non-V ocational High 
S chool Gr aduates in TeKas, John A. Laska 
and Jau-Woei Chlou, 1973* 

The Cont ributions of Vo cational Education, Training 73 
and Wo rk Exp erience to the Early Career AQhieve - 
. mahts of Young Men * John T . Oraiso , "r97 5 ," 

A Decade of Feedback Information from Harcum G radu- 77 
" " atesi 19"6 8-1977" ". "Boris Elai, Jr.", 1977. 

Effectiveness of Voca tion al and Technical Prog rams i 75 
A "Natio nal Follow-U p Survey / Gerald G* Somers" 
and others , 1971 * 



Complete bibliographic citations for these titles are 
provided with the annotations on the pages that follow. 

53 



63 



Page 

Employ er Satlsga cfclo n with the Skills of Vooatlorial 82 
Education Graduates In Nort h Dakota^ Larry 
Smlleyi 1976 .' ^ " " 

"A Follow-^up Study of Former Occupational'-Technical 85 
Students at Virginia Community Colleges." Edith 
H. Carter^ 1076. 



A FQllow-up St ud y of Illinois Home E coriomics Job 

Training PrograinsT Kathleen llov/ell and Joyce 
L. FalstehausenV 1P71. 



07 



h Fq11o\^-up Study of M achine Tool T echnology and Build - 89'' 
ing Construction Graduatea", Wilbur Hall / Rodney 
dray^ and Arthur 0. Berry/ 1975 

Follow-up Study of 19 69-1975 GraduateB of the Plvl- 91 
sioii of Te chnology of New Yo rK dty "Community 
CQllageB . Jeanne H. "Gammel, Stanley V''^ Grodsky, 
and Richard Alfred ^ 1976. 

"A Follow-up Survey of the Graduates of the High 94 
Schools in Vocational Region #8/ New Hampshire," 
Boger D* Crim and Eugene Rosis ^ 1''7^. 

T he Impact of Second ar y School Occupatlon n.l Educa- 96 
~ tion In MaosachusettsV " William ^ . nonroy^ Jr. 
and Daniel B." Diamond^ 1976 , 

"Intensity of Occupational Training! Its Effects on 98 
Subsequent Labor Market EKperiences , " Robert E. 
Allen and Thomas G. Gutter idge, 197 0- 

Job Selection Patterns; Linkage Be t^/^een ^ ^ocational 101 
Education rrograms an d the" Labor :"ar]^.Qt, George 
H. Copa and E3ruce A." Kleven^ 1977 . 

"Little ??oolly Horses and Saber-Toothed Tigers s The 10 4 
Misguided Vocational Education Curriculum*" 
Everett Egginton^ no date* 

1975 Career Student Follow-Up; Initial Placement . 10 S 

^ "Michael Quant// 1976* 

lOOO Employe rs Look at Occupational Education. 110 
Martin Haraburger and Harry E . "Wolf son/ 1969 . 



EKLC 



54 

64 



Quteomes of Vocational Education in Virginia 1978 i 
A SummRy of the Follow-up q1_1167-^7 7 Secondary 
VQcatlQnal Eduaation CQmplitions in Virginia , 
Donald E. Elson, 1978- 

"Outconies of Vocational-Technical-Transf er Programs 
at Conimunity Colleges^ Technical Schools^ and 
Similar Types of Institutions.-* Richard J. Noeth 
and Gary Hanson, 1976* 

A Program Review of Secondary Vocational Educ ation 
"~ in Oh ioV Job" Placement and State Funding . 
Ohio Legislative Services Commiasion/ 1978 » 

A S tudy of Community C ollege Students Who Are Grad - 
uates of Vocati onal "Technical and College Prep - 
ar? Eory Curriculum, Ronald J. Horvathf 197 3. 

A Study of the Costs, B enefits, and Ef f ectivenesa 
of Occupationa l Education. Austin Swanson, 
1976. 

Study of the Status a nd Effectiveness of Cooperative 
Of^fioe Education in New Jersey f968-69. Carmela 
Kingston, 1970. 

"The Success of Vocationally Trained Women in Tradi-- 
tionally Male Occupations J' David J. Pucel, 1974 

Vocational Education Planning District s in Ohio: An 
EconQmlc Evaluation o f Foregone Benefits ttom 
Limited' Participation. I. A. Ghazalah^ 197 5 * 

Vocational Technical and Adult Education: Student 
Follov^^ Jim 
PrestonV 197 6 , 



55 



A-28 



HI. EMPIRICAL OUTCOME STUDIES 



B, Citations and Annotations 



The Adequacy o f VQcatlonal and Technical EducatlQn , South 
Carolina Advisory Council on Vdcational and Technical 
Eduaation* Coliambiai South Carolina Mvisory Council 
on Vocational Education^ February 1975^ 46 pp. 
(ID 126 315) 



DESCEIPTIVE ANNOTATION i 

The South Carolina Advisory Council on Vocational and 
Technical Education culminates its evaluation responsibili- 
ties each year by preparing and publishing an annual 
evaluation report/ It became apparent^ however, that ''If 
one is to adequately evaluate the * effectiveness ' of programs^ 
what better perspective eKists than that of the employers?" 
This study was initiated to eKplors that often-neglected 
perspective. Many questions were asked i 

0 Are technical and/or vocational centers a 

good source of trained, prospective employees? 

m Do the centers meet the needs of businass 
and industry? 

• Do they meet the needs of the students? 

m How do graduates of the centers compare 
with other employees? 

m What is the quality of this education? 

Recommendations concerning any aspect of the centers were 
sought as well* 

Ten thousand employers of large companies within the 
state constituted the population for the study; question- 
naires were sent to each of them* Usable returns were 
received by the cutoff date from 1161 employers, a response 
rate of 12%* Characteristics of the respondents are dis- 
cussed and presented in tables* A cross-sample of the 
South Carolina business and Industrial community and of the 
geographic regions of the state were represented. The 
survey form was mailed in October 1975. 'The questionnaire 
does not ask when employers had eKperience with graduates of 
vocational and/or technical centers. 
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The survey questionnaire, drawn up by a consulting firm/ 
was designed to answer the basic questions the advisory coun- 
cil had raised* After review by a committee of the council 
and by representatives from vocational and technical educa- 
tion, it was revised. The instrument is appended. 

The data are presented in tabular form; comparisons 
are sometimes made between all respondents and those respon- 
dents who have hired vocational/technical graduates. Some 
data are presented as frequencies and percentages of employers 
giving particular responses to specific questions; others 
are in the form of average responses and rank orders* Some 
of the recommendations are listed with the percentage of 
respondents maKing each recomitiendation* 

.Ajnong the con: iusions discussed are the following i 



• Employers consider both vocational and 
technical centers a good source of trained 
employees^ generally rating those employees 
higher than others, 

0 Both vocational and technical centers meet 
the needs of business and industry and of 
the students^ but excel at meeting the needs 
of students. 

• Employers are more familiar with technical 
education centers/colleges than with area 
vocational education centers* 

m Vocational and technical programs are 

oriented more to manufacturing than to other 
categories of business and industry* 



Specific recommendations are made to the State Board 
for Technical and Comprehensive Education^ to the State 
Board for vocational Education, and to the State General 
Assembly* Included are recommendations for continued fund- 
ing and for greater dissemination='-to the public-^^of 
information regarding available programs. 

This study was initiated by the South Carolina Advisory 
Council on Vocational and Technical Education, The Dunn 
and Bradstreet^ Inc, , Marketing Services Division was hired 
because it had access to the desired population (through 
computer files of addresses) and because the council had 
confidence in its researGh capabilities* 
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EVALUATIVE COMMENTS I 

This study is useful in an exploratory sense because it 
suggests areas for further research. The low response rate 
makes such research necessary before findings from the study 
can be more than suggestive. The additional research should 
focus on different occupational areas. For eKainplef do em^ 
ployers of trade and industrial students view vocational edu- 
cation differently from employers of business students? 

Another question that can be raised from this study 
concerns the extent of experience employers have had with 
vocational students and what effect^ if any, that has on 
their perceptions. If employers? have hired only two students 
each/ one would expect their perceptions to be less indica-- 
tive of the results of vocational education than the percep- 
tions of those employing hundreds of vocational students - 

The researchers selected a large sample hoping to 
obviate bias both from respondents with veeted interests 
in the program and from those who tend to give answers they 
believe desired. In further research , an attempt should be 
made to assess response bias^ and technlgues of question^ 
naire construction should he used to minimize some kinds of 
bias. For example, a forced-choice format can be used to 
prevent respondents from giving "desirable" responses. 

One conclusion drawn in this report is that although 
vocational and technical centers meet the needs of business 
and industry / they better meet the needs of students^ This 
issue may deserve further study. Some questions are 
obvious % 

m What are the policy implications of this 
statement if it is a widely--held employer 
perception? 

• what are the implica-i=ions if it is indeed 
"true"? 

• Are the implications such that the issue 
should be studied in- depth, i*e., which 
programs meet which needs of which employ- 
ers and which . students? 

If further research is warranted, one aspect of that research 
should be a study of student perceptions as compared to 
employer perceptions * 

* * * 
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Animal Health Technicians: A Survey of Program Graduates 
.. ~ aind of Veteri narianB , " Richard B* Barsaleau and Henry R. 

Walters, Sacramento i Consumnes River College, March 

1977, 16 pp, (ED 136 891) 



DESCRIPTIVE ANNOTATION! 

Consumnes River College wished to assess the impact of 
its two-^year Animal Health Technician (AHT) program. Two 
studies were conducted to investigate the employment status 
and earnings of AHT program graduates i a survey of the 
graduates and a survey of local veterinarians, the employer 
group most likely to hire AHT graduates* A question of 1 
special interest addressed only to veterinarians asked j 
whether graduates of any accredited AHT programs were 
"accepted" by the majority of local veterinarians* 

At the time of the graduate study, three classes of 
trainees had been graduated and 6 8 students had received 
A*S. degrees. Of these, 47 responded to a mailed question^ 
naire (a response rate of 69%): 44 were female, 34 were 
unmarried^ and the age range was 20 to over. 35 with 34 
under 25* Although development of the questionnaire is not 
discussed nor is the instrument appended, a list of the topics 
surveyed is provided. Included were employment status, job 
stability/ and earnings. The report included the following 
findingsi 1) 33 or 70% of the respondents were employed as 
AHTs, 14 or 30% were employed in related fields and one 
.was unemployed by choice; 2) 15 AHTs out of 22 tallied on 
this issue were spending 25% or less of their time in front= 
office duties, clerical work as opposed to animal care work. 
The data are not statistically analyged. 

The second survey was mailed to 139 licensed California 
veterinarians (most in the Sacramento Valley) ; 71 responded, 
yielding a response rate of 51%. Sixty-two of the respon-- 
dents employed animal health technicians, of whom 27 were 
graduates of an accredited AHT program, 34 had been trained 
on the job, 1 trained in a high school Regional Occupational 
Programi, and 5 trained in proprietary schools. The authors 
state that the practice of on-the-job training is diminish- 
ing as more trained AHTs are becoming available* The devel- 
opment of the survey questionnaire is not described nor is 
the instrument anpanded , but issues covered by the survey are 
listed, 

ResultiS caiscussed include the following s 



• Thirty-seven responding veterinarians 
indiaated a willingneBS to provide Work 
Experience Stations for AHT students. 

• From 47 to 60 veterinarians checked training 
needs in the areas of identification of 
animal breeds / laboratory techniques^ 
reception procedures^ and hospital safety. 

• Some respondents commented, "All the Consumnes 
people I have met seem quite well informed 
regarding specific practices and their 

modes of operation" and "The only real 
problem I have jtseen ^is overconf idence 
which can lead an AHT to give misinform 
. mation to a client , , * 

The study concludes? "Consimnes River College graduatae 
are making their marks and are finding willing employers 
who will use their training and talents in a legal and 
ethical manner." 

The surveys were an in^house project conducted by the 
AHT Program Director, C, Barsaleau, D.V,M, ^ and 
H. R, Walters of Career Education/Institutional Research, 
both of Consmnnes River College. 

EVALUATIVE COWMENTS i 

The major purpose of this study was to determine whether 
veterinarians in the Sacramento Valley area were accepting 
graduates of accredited AHT programs. As most graduates of 
Consiimnas River College's program were employed as AHTs and 
surveyed veterinarians gave a range of positive responses 
concerning the programs, a degree of acceptance can be 
inferred. How much is not clear since isisues such as possi-^ 
ble response bias are not discussed. The study has greater 
potential than was realised, however, or at least than was 
reported- Although some responding veterinarians were 
employing program graduates and others employing technicians 
who had received their training in other ways, no compara-- 
tive data are reported. 

Because the AHT program is not described, nor, is 
aevelopment of the questionnaire described or the instrument 
appended^ results of the study cannot be applied to other 
AHT programs in other areas nor can the study be replicated 
from the information given in the report. 



An Assessment of Benefits Derived from Membership in a 

VQcational Student O rganiz atlQn "in the Vocat Tech 
nioal and Adult Education System . "Dennis R,' Collins. 
Menoininees Center for Vocational^ Technical and Adult 
Education/ University of Wisconsin-"Stout , August 1911, 
92 pp, (ED 145 234) 



DESCRIPTIVE ANNOTATION: 

The primary purpose of this study was to deterrnine 
whather participation in vocational student organizations 
(VSOs) results in leadership ability / career growth, 
comiTiunity involvement , and social development on the part 
of participating students. It was hoped that positive 
findings would lead to greater emphasis on VSOs in the total 
spectrum of vocational education curricula. Neither specific 
research questions nor hypotheses were stated. 

Four populations were identified for this study i 
(1) state directors of vocational education; (2) advisors 
of VSOs; (3) Wisconsin post^secondary VSO menibars; and 
(4) highly participating Wisconsin VSO members. The first 
two groups were questioned about proce:Ss and structure 
variables and so will not be considered further in this 
annotation. The VSO member sample consisted of 15% of 
post-secondary members of thirty chapters of VSOs in Wisconsin 
including the Wisconsin Office Education Association, Dis- 
tributive Education Clubs of America (DECA) , Vocational 
Industrial Clubs, of America, VJisconsin Association of Health 
Occupations, and Wisconsin Home Economics Related Student 
Organization. The response rate was 5 6% of the sample. The 
sample of highly participating VSO members consisted of 
33 DECA members attending the Wisconsin DECA Junior 
Collegiate Career Development ConferenGe in Madison in 
March 19 76. 

Written questionnaires were developed for this study 
and samples are appended to the document. The guestiorinaires 
were mailed to VSO advisors who were instructed to administer 
them as randomly as possible to 151 of the members. Students 
used a 5-point Likert-type scale to rate the benefits of 
VSOs in terms of certain outgrowths i 

m leadership characteristics---ability to 
speak effectively/ to make decisions/ to 
exercise creativity 

• career growth — pride in work/ awareness 
of job qualifications/ understanding of 
the effects of career choice on life style 
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m conmiunlty awareness^-- involvement in 
political activity and social affairs 
of general interest 

• social developiTient^-^teaniwork participa- 
tion/ positive. self-imag€f ability to 
make and keep friends 

The data are presented in f requency/peroentage tablis 
and in graphs^ Data from tfie highly participating students 
and the randomly selected etudents are presented side-by^ 
side for comparison piirpoBea but no statistical analyses 
of differences have bean made. Ratings of four or five 
(1 ^ nona^ 3 - moderate^ 5 =- significant) were considered 
■to Indicate that benefit was derived in terms of the 
characteristic rated. 

Discussion of these findings are included: (1) forty-- 
eight per cent of the participating members gav^ two 
leadership characteristics a rating of four or Livm, arid 
a larger percentage gave such ratings to all other leader-- 
ship characteristics; and (2) the responses of the randomly 
selected members were not as favorable as those of the 
highly participating members, with ratings of four or five 
being made by only 40-SOi of the resporidents for most of 
the leadership characteristics. 

The author makes several re commendations for giving 
greater emphasis and support to vocational student organi-- 
zatlons . 

This study was conducted by a graduate assistant at 
the University of Wisconsin—Stout, Center for Vocational, 
Technical, and Adult Eduoation pursuant to a grant or con- 
tract with the Wisconsin Board of Vocational, Technical, 
and Adult Education. 

EVALUATIVE COMMENTS i 

Although the stated purpose of this study was to 
enipirically investigate claims regarding the benefits of 
voeational student organizations, the data collected toward 
that end were strictly opinions of students. No justifica- 
tioii for the assumption that students have the ability to 
assess and report those benefit! acourately was given. 
Further, since the researcher did not randomly select 
meitoers, it is not clear that the respondents to the 
guestionnaire constituted a random sample of VSO members * 
For these reasons alone, it is difficult to come to any 
conclusions about the "actual" benefits of participation In 



The rating scale used is positively biased. It is 
different from most Likert=»type scales tn that the left 
andpoint is '*none" and every other point is positively 
defined^ whereas the usual scale is symmetrical with two 
or niore negative points, a lero point, and two or more 
positive points* How responderits interpreted this scale is 
not clear. Further/ although the author interpreted ratings 
of 4 or 5 to indicate a derived benefit, laws of probability 
predict that 40% of respondents would indicate a benefit 
derived (since, given five choices, random selection would 
result in each choice being selected 20% of the time) , The 
author provides no rationale for his assumption that 
positive ratings by 48% or more of the respondents is dif-- 
ferant from chance eKpectation and for the assujnption that 
it indicates VSOm as effective in leadership development. 

The analyBis ^oC the data was inadequate. Although the 
author compares re^.ponses of randomly selected inembars and 
highly particvt^at: Ing members, he uses no statistical tech-- 
niques to determine whether differences in responsas ware^ 
different from those expected by chance. A statistical 
analysis such as a chl-^sguare could have been used to 
advantage. In addition, a more serious problem exists. It 
is possible that observed dlffarances between the two groups 
are functions of "self-selectlun" rather than of intensity 
of participation in a VSO. Xt is possible that factors 
causing students to be highly participating also cause them 
to develop leadership characteristics or to be better able 
to. These possibilities are not acknowledged, 

* * * 



Benef it "^Cost Comparison of Vocational Education Programs. 
Statewide Evaluat ion of Vocat ional-Technicai Education in 
Florida-^_Volume 2. Marshall A Harris , Tailahassee"i ~ 
Florida State University, Florida State Advisory Council 
on Vocational and Technical Educationjr 1972/ 118 pp. 
(ED 074 223) 



DESCRIPTIVE ANNOTATION^ 

The general purpose of this study was to determine 
whether vocational education in Florida has sufficient 
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economio payoff to bs GOnsidered a favorable medium for 
invasting both public and private (student) resources. 
Methodologically, it builds on and improves upon two previous 
Gost--ef feotivenasa studies conducted in Florida, Three 
questions are addressed! 

• Do vocational education programs in 
Florida have positive benefit-cost 
relationships? 

• UQ benefit-cost relationships between 
programs vary and do these relationships 
vary from student to student? 

4 can the results of a benefit-cost analysis 
be useful to educational planners and 
decision-makere and to individuals antici- 
pating enrollment in vocational education? 

The population included persons who either coinpleted 
training or left early during the period from August 196 8 
through September 1971* The state was divided into four 
geographic regions^ and two Area Vocational Centers were 
selected from each of the regions^ with inclusion based on 
enrollment and program factors. When a random sMiple of 
fornier students had been selected^ questionnaires were 
mailed* Thres hundred eighteen usable questionnaires were 
collected from the 820 mailed^ a response rate of 391. 
ResponBes from engineering students were excluded from the 
study £i0 were those from former students with no training-^ 
related job eKperience* A table indicating numbers of 
usable returns and reasons for nonusability by vocational 
programs is provided. 

The mailed questionnaire asked 16 general questions as 
well as specific questions about each job held (title ^ 
earnings^ etc). Portions were adapted from a quastionnalre 
developed and used by other researchers,^ The questionnaire 
is appended. ' . 

Both public and private costs and benefits are analysed ^ 
and technical aspects of the analyses are discussed in the 
body of the report and in appendices. Chi-'Squares^ Scheffd' 



"Jacob J, Kaufman^ Ernst W, Stromsdorfer , Teh^Wei Hu^ 
and Maw Lin Lee, A Cost-Effectiveness Study of Vocational 
Education i A Comparison of Vocational and Non-V o cational 
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testS/ analysis of variance, and multiple regression are 
used to analyze the data. A benefit^cost planning model is 
presented as well. 

Among the six "most cogent findings and conclLisions" 
discussed are (1) * .on the average, society will recoup 
its average investment of $1|716 per student in 1,3 years 
and on the average, a student will recoup his average invest'- 
ment of $2,411 in 1.9 years" and {2) "On both public and 
student investmentB , nonsecondary students had higher rates 
of return than did secondary students*'' 

EVALUATIVE COMtffiNTS i 

This report is notable for being very detailed, Cost^ 
benefit analysis is a complex technique, and detail is a 
necessary prerequisite for accurate assesnment. However, a 
discussion of the pitfalls of this type of analysis can be 
found in Attachment "C," and findings should be viewed in 
light of those pitfalls. 

A strong point of this study is that a questionnaire in 
a previous study was adapted. If a knowledge base about voca=^ 
tional education is to be built, some comparability in data 
collection and analysis are necessary. The repeated adminis-- 
tration off the same, or at least similar, instruments is one 
method of achieving this comparability, 

* * * 



Coinparative Analysis of Post-Secondary Qgcupational and 
Educational Qutc omes f or the High "SchoQl'Class of ~ 1972 , 

Reid Creech, Norman Freebe"rg~, Donald~A* RocR;, 
Kenneth M* Wilson, and Kan-^Hua Young, Princetons 
Educational Testing Service, May 1977, 465 pp* (ED 139 
845) 



DESCRIPTIVE ANNOTATIONS 

This report is the result of a project having two 
objectives I "to effect a partial evaluation of the effec»' 
tiveness of vocational education as compared to academic 
and general high school programs" and "to develop informa-- 
tion useful for program-planning specialists involved In 
vocational education, " To accomplish these objectives^. 
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various aspects of "the current^ status and activities of 
the claaa of 1972" were investigated, Iriclufling educatiohal 
activities, job changes^ duration of employment, job satis-* 
faction/ and weekly earnings (Chapter 2) , Also investigated 
was "the realisation of plans and aspirations" (Chapter 4). 
Sonie chapters of the documerit describe the social makeup 
of the high school class of 1972 ^ the xrariables used in the 
studyf and the methods of analysis; others discuss "non^ 
response and related concerne" and "Where go v^e go from 
here?" The major findings of the study are swnmarised in 
Chapter 5. . 

This study is based ori data collected in the Natiohal 
Longitudinal Study of the High School Class of 1972,3 one 
of a series of National Longitudinal Stiidies of Educational 
Effects sponsored by the Departrnant of Healthy Eduqation , 
and Welfare* Based on both the original set of data and on 
the first 18-month follow-up^ it "represents the first 
in*-depth investigation of what has happened to the Class of 
1972 since its graduation, with speciaL emphasis on educa- 
tional and vocational outcomes," 

Nearly 3,000,00 0 students graduated from high schools 
across the nation in 1972* About 18,000 of these students . 
were selected, according to a carefully designed and 
exacuted sampling plan, to participate in the National 
Longitudinal' Study of the High School Class of 1972, 

The development of the first follDw--up of questionnaires 
(FFQ) ^ which was usad to collect much of the data discussed 
in this report, is not described in this document. A form 
of the questionnaire is appended however and the question- 
naire items are discussed, Nonresponse and related concerns 
are also discussed. 

The data analysis techniqueB employed in this study are 
thoroughly described* Multiple polnt-biserial regression 
analysis was used for comparisons between two groups, and 
multiple discriminant analysis was used for comparisons 
involving three or more groups* An econometric model using 
a simultaneous two-equation system was used to examine the 
interaction between college attendance and work participation 
Five structural (path) 'analytical models were built, and 



The time period referred to as current roughly repre- 
sents May 197 2 « January 197 4* - , 

^Mm^ ann^nfli'^ nf K^\/ Related Tilf^rature for a CI ta- 



tested in an effort to assess what underlies decision making 
in the ocoupational--educational aspiration area, post-high 
school educational progress* 

Among the results discussed in the "Summary of Major 
Findings" are these: 

• At the time of the first follow-up survey, 
64% vere employed and 8% were out of work. 

• The employment rates for graduatea of vari- 
ous curricula were vocational i 77%; generals 
68%; and academic; 56%, 

m The homemaking rates ^were vocational i 4 0%; 
general! 36%; and academloi 15% « 

• Earnings were similar for blacks and whites 
but much higher for males than for females. 

• On the average, vocational graduates 
worked a greater number of weeks from 
October 1972 to October 1973 than did 
graduates of other curricula , 



The project investigators were from the staff of 
Educational Testing Service. The project was oonducted 
pursuant to HEW Office of Planning, Budgeting,, and Evalua- 
tion Contract' No, 300*^75"-O312 , a response to Office of 
Education RFP 75-50. 

EVALUATIVE COMMENTS: 

Time and space do not permit a thorough analysis of 
this document. It ^s notable in that the variables and 
methods of analysis/ are described in detail; however, the 
reader is cautioned' to thoroughly review both this document 
and other documents describing the National Longitudinal 
Study of the Class of 1972 before accepting any of the 
results listed here. 



* * * 
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A Cofliparatlve StTad y of the Occupational Aahievement of 
Vocational and NQn-Vocationa l High ScHqoI G raduates in 
the "State oJE Ar kansas . " Dean C/ Andrews and Lawrence H., 
Roberts* Magnolias Educational Plannlno- and Evaluation 
Services^ Deceiaber 20, 1974, 131 pp, (Efi 112 207) 



DESCRIPTIVE ANNOmTlON i 

This study was a response to the aoncerns of taxpayers 
and deciaion-rnakers in regard to the effectiveness of edu^ 
eatipn. Specif ically^ it was aonducted to acquire feedbaak 
information to be ueed in improving the vocational educatiori 
progrania in Arkansas. A ^ost-benefit study vras' considered 
but rejected because ^ among other reasons^ the cost would 
have been prohibitive and too many methodologiaal issues 
are yet to be refined. Instead^ this study sought to 
compare the occupational achievernents of students who were 
anrolled in vocational education with those of students who 
were not so enrolled with respect to their (1) occupational 
status^ (2) educational status, (3) earnings, (4) job 
satisfaction, and (S) other related factors* 

The study ^ conducted in 1974^. had as its population the 
1970 graduating classes of eight Arkansas public high 
schools* Three sajtnples were selectedi (1) those who had 
had two semeBters of vocational classes; (2) those who had 
had four semesters of vocational classes; and (3) those who 
had had no vocational training* In order to ensure that ' 
the third group was comparable to the first two, those 
graduates who had had grade point averages (GPAs) above 2.25 
and had planned to go to college were eliminated. A total 
of 1749 of the 259 7 graduates of the eight schools were 
sent questionnaires yielding an overall sample of 67%, 
Three mailings were used in an attempt to reach as many 
graduates as possible; the final response rate was 36% of 
the sample (2 4% of the population) . Tables indicating 
responses to the separate mailings and response rates by 
sex^ race, GPA, and vocational training are provided, 

A preliminary guestionnaire was developed and pilot^ 
tested for clarity and reading level with eleven 1973 
graduates of one of the high schools used in the study. 
Those graduates made suggestions that were incorporated into 
a revised form subsequently administered to five more gradu^ 
atef>* In addition, several professional educators critiqued 
the preliminary questionnaire, A oopy of the final form is 
provided in an appendix. The data from school records and 
the guestionnaire s were summarized on pre-designed tables 



descriptive— summary tabulations of frequencies and percent- 
ages. Chi-aquara analyses were used to determine whether 
statistically "sigrnificant differences existed between 
categories. 

Topics discussed in three chapters include the ogcu- 
pational activities of the vocational graduates, the 
vocational graduates' evaluation o£ their training^ and 
a comparison of occupational achievement of the £ our--Bemester 
vocational graduates and the nonvocational graduates. A 
subsequent chapter sumnarizes the results and provides 
conclusions and recorraftendations . 

Chi-equare analyses were done to compare the vocational 
and nonvocational groups on the basis of sexual compositions- 
racial composition; and grade point average. The only 
statistically significant difference between the groups was 
in grades: a higher proportion of vocational graduates had 
GPAs above 2*00. The authors acknowledge that the results 
reported^ of which the following are a sample, should be 
interpreted in light of this grade difference i 

m I'here are no significant differences in 
occupational status between graduates of 
each group (i.e.; no significant difference 
in proportions of those working full-time^, 
of those attending college ; etc*)* 

• A greater proportion of vocational graduates 
are employed in trade and industrial areas. 

• There are no significant differences in the 
degree of job satisfaction expressed by the 
vocational, and nonvocational graduates. 

m There are no statistically significant 
differences in earnings i 

■ 

Based on differences in occupational status between 
vocational students with two semesters of vocational courses 
and those with four semesters , the authors conclude that 
.two years of vocational training is desirable for those who 
plan to work after high school. Based on the lack of 
relationship between type of job and type of training , 
however I they reconmiend that local education agencies study 
the relationships of their offerings to the job market and/or 
implement a vigorous placement program. Still; they conclude; 
vocational graduates have a greater chance than nonvocational 
graduates of finding full-time work they consider closely 
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This studly was aonducted by two members of ■'Educatibnal 
Planning and Evaluation Services^" based in Magnolia, 
Arkansas* 

EVALUATIVE COMMENTS i 

This study i a notable for at least three reasons i 

m The authors indicate that they chose a 
methodology in light of their purpose. 

m The data-collection instrument was 
pilot-^testedt 

^ • Dif fetencee between groups were 
statletically analyzed. 

Because graduates who had four aemeeters of vocational 
training had a higher occupational status than those with 
only two semesters^ the authors conclude that four semesters 
are better than two. They failed, however, to consider the 
possibility of a self -selection factor i Perhaps those 
students who completed four Bemesters have other qualities 
helping theni achieve a higher occupational status than 
those with only two semesters. There are a nimber of ways 
this cjuestion can be explored in further resaarah* 

* * * 



h Comparative S tudy of the Occupational Achievement of Voca - 
tionai and Non-Vocation al High Gradua tes in Texas ^ 

John A. Laeka and "jau-^Woei Chiou. Austin i Department of 
Cultural foundations of Education ^ Center for International 
Education, University of Texas at Austin, June 30^ 1973, 
183 pp, (ED 118 944) 



DESCRIPTIVE ANNOTATION I 

The primary purpose of this study was "to determine 
whether there were any real differences in occupational ^ 
achievament between the high school graduates who had been 
enrolled in vocational education programs and the graduates 
..1 w^^^^j. T^^*^ 1 «^ ^ f^hmm^ nrnCTrams." Tt warn intend* 
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4pC:;«i:^sted taxpayers, legislators, and political leaders] to 

about the efficacy of . . . large expenditures in 
f^P^ph^ing students for gainful employment in the labor market." 
.rurther/the findings of this empirical research were intended 
to be used for continuing improvement of vocational education 
programs^ 

The literature review section cites the Little review^ as 
wall as some specific studies, "In suiranary, a review of exist- 
ing research indicates that earnings f length of employment, 
job satisfaction, occupational status, and occupational mobll-^ 
ity were commonly used as the indices of occupational achieve- 
ment. Soma other variables were found relevant in accounting 
for these indices of occupational achievement. They were seK, 
ethnicity, academic achievement, social origin, college 
attendance, community differences, and vocational education. 
The findings obtained from existing research were eKtremely 
useful in developing the conceptions and in deciding the 
scope of this study," 

For this study, vocational students were defined as those 
who had completed two credits in a school year in a vocational 
program. Non-vocational students were matched to vocational 
students according to sex and accumulated scholastic ranking. 
Complete information on the sampling procedure is provided* ■ 
The population consisted of students who had graduated in 1970 
from 17 public high schools in 5 independent Texas school dis- 
tricts. A letter and questionnaire were sent to 3,045 gradu- 
ates in August 197 2. Follow-up mailing was sent in September^ 
In all, iOSa (3SU the 3, 045 graduates responded to the, 
questionnaire, 5^%: could not be reached, and 148 5 (49%) 

either indicatt^d: k reiusal to cooperate or did not return the 
questionnaire by the October deadline. Response rates by 
schools are tabled as are characteristics of the sample. 

A survey questionnaire was designed to collect data from 
which to derive six indices of occupatiQiial achievements. 
Three indices concerned income-^aQCumulated income, monthly 
earnings^ and hourly rate-^^and three concerned employment- 
weekly working hours, months of employment, and actual job 
length. In order to account for possible spurious, but sta- 
tistically significant, relationships, several control vari- 
ables were introduced, including sex, community, ethnicity, 
social origin, marital status, college attendance, and aca- 
demic achievement. Other possibly relevant variables, such 
as cguality of teaching personnel, teaching facilities, work 
motivation, personality traits, and labor-market situations, 
were defied beyond the scope of this study. 
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The questionnaire was pilot-tested extensively to ensure 
that it wae comprahensible . Item anal^; ses were perfornied to 
eliminate redundant itOTSt The survey questionnaire is ap^ 
pended as are coded raw data. 

The two major statistical techniques used in this study 
are analysis of variance and multiple regression analysis* 
Factor analysis is used to Interpret the interrrelationships 
Miong different indices of occupational achievement. The 
rationale for each specific analysis used is thoroughly dis- . 
cussed and several tables of data are provided. Results of 
preliininary analysis and subsequent analyses are thoroughly ; 
discussed with accompanying tables. 

Among the conclusions and policy Impllaations (based on 
894 respondents) provided in the final chapter are these i 

• 244 completed two or more years of college/ and 157 
attended college from 4 to 18 months during the 27 
months after high school graduation. For those stu- 
dents then working i secondary vocational education 
was not an important variable in initial employment 
when community/ sex^ and college attendance v/ere con- 
trolled. 

• The affects of secondary vocational education were 
statistically significant (.05 level or beyond) in 
accounting for the accumulated income and actual job 
length of the 493 non-college workers of both sexes > 
the months of employment of male non-college workers, 
and the weekly working hours of female noR^^oollege ^ 
workers in the Austin-San Antonio area* not in 
Houston* 

m Community factors seemed to relate to the operation 
of secondary vocational education in different ways* 

• Beneficial returns of secondary vocational education 
were definitely real and substantive in terms of 
yearly accumulation. 

The researchers are members of the Department of Cultural 
Foundations of Education, Center for international Education , . 
University of Texas at Austin, The study was conducted pur- 
suant to a contract with the Department of Occupational .Edu- 
cation and Technology of the Texas Education Agency^ 



EVALU^VriVE COMMEOTSi 

Thie is n eompleK study with several notable features, 
Ratiora^j^ based ij^ part on a literature review^ for the 
choseii variables was provided; the survey instrument was pilot- 
tested; the data analyses are thoroughly described; and the 
raw data are appended to the report* It is one of the few 
studies in which vocational students are compared to non^voca- 
tional studentB. The students were matched according to sex 
and scholastic ranking. However ^ the authors do not acknow- 
ledge" the limitations of matching as a tool in quasi^eKperi-- 
mental research. 



Tha^ Contri butions of VQa ational Education^ Training ^ and Work 
Ekperlence to the Early Career Achievements of Young Men . 
John Tp Grasso, Doctoral dissertation presented to the 
Faculty of Educational Development of The Ohio State Uni=' 
varsity, Columbus, July 1975/ 167 pp, (ED 113 537) 



DESCRIPTIVE ANNOTATION: 

The author states "Much of the decision making underly- 
ing vocational education legislation and expenditures occurred 
in the absence of adequate inf orination . " He goes on to 
say that althaugh research has been done and some of it is 
encouraging in its diversity of perspectives/ the policy rele- 
vance of the research "has been circumscribed by limitations 
inherent ih the design of many of the studies, or as a result 
of conceptual br methodological difficulties/ often compound^ 
ed~--if not caused-=by a lack of adequate data." Other limi-- 
tations are discussed as well. 



This study was conducted to contribute to "improvement 



• attitudes toward the adequacy of their 
preparation for work 

• participation in post-^school training 
and learning activities 

• skill level 

• wages earned 

• job satisfaction 

• unemployment experience 

• long-^run career prospects 

Except when noted, the aonclusions drawn in this stud^ 
are based on a subset of the National Longitudinal Surveys^ 
(NLS) sample I those young men not enrolled in school at the 
time in question who had completed high school but not even 
a year of college. "A national probability sample of over 
5#000 males between the ages of 14 and 24 was interviewed , in 
the last quarter of calendar year 1966 and annually there-^ 
after through 1971; inforniation collected through 1969 was 
available for this study The young meTi were divided into 
four curricular categories i college preparatory / general^ 
vocational^commercial ^ and vocational=otner (referred to as 
vocational) * 

This study is based on the personal interviews and the 
1968 special mail survey conducted as part of the National 
Longitudinal Surveys. The in.-^trumentation is described in 
various NLS documents* All analyses in the study are based 
on data statistically weighted to allow reliable comparisons 
between black and white youth. Further delimitation of the 
youth on the bases of vocational program studies or extent of 
study were not possible; however^ it was possible to statis-- 
tically control for the following demographic variables s 
socioeconomic status ^ scholastic aptitude r and race . Other 
control variables used are price level and demand^ military 
service/ residence at age 14 (rural farm, rural non--farm, 



A-47 



The results relative to the eight factors itemized in 
the first paragraph of this annotation are described in de- 
tail with several summary tables- The following are included 
among the findings: 

» Neither vocational nor vocational^-commerical 
students possess greater occupational infor*- 
mation than general track students. 



• Vocational graduates do not consider thc.^i-^ 
selves better prepared for the world of 
work than general graduates* 

• Many graduates of the various curricula 
sought further trainings-college^ pre- 
professional or technical ; commercial , 
managerial; vocational^ skilled manual • 

• There is no indication that vocational 
graduates obtain jobs with higher sitill 
requirements than other graduates, but 
thOBe with past-high school training (ex- 
cluding college) obtain jobs with higher 
ratings* 



• In terms of wages, white vocational students 
benefit more from post-high school training 
(excluding college) than other white students. 
However^ there were no wage differences found 
among black youth '^^om different curricula. 

• Job satisfaction Jlts reflect the earn- 
ings results repc tf-:! vn the preceding item 
for white youths^ but again no curricular 
differences were found amo: black youth. 

• No conclusions on unemployment eKperience 
with respect to curricular differences 
could be drawn from the data. 

m College preparatory students and white commer- 



The final chapter presents conalusions and policy impli- 
cations derived from tha. study. There ar least four 
major implications for secondary school 5 ixcy makersi 

• Schools need to impart more information regard- 
ing careers in general and about the role and 
importance of pDStsecondary school training and 
learning opportunities. 

• The accessibility of posteecondary training 
programs should be increased^ 

• Additional training for black youth must be 
promoted^ but efforts must be accompanied by 
affirmative action labor -market policies* 

m Speoific types of career^^ taest served by each 
curriculum should be asnertained and programs 
modified accordingly to serve the needs of youth 
and society* 

This study was iindertaken, in part, to fulfill require- 
ments for the author's doctoral degree. He spent five years 
working on the National Longitudinal Surveys project while 
completing his doctoral studies. 

EVALUATIVE COMMENTS: 

Several limitations of the curriculum variable are acknow^ 
ledged by the author* Occupational programs, for example, 
auto mechanics and agriculture, are not distinguished^ stu-- 
dent reports of their curricula were not confirmed; the ex- 
tent or intensity of vocational study undertaken in high 
school was not noted; and students who took a "general" cur- 
riculum with a practical arts emphasis (for example, general 
business courses) are not distinguished froin those who took 
a general curriculum without such emphasis. In spite of its' 
limitations, however, this study with its statistical con- 
trols of several variables is an improvement over several 
less sophisticated studies. 



A D^jcade of Feedback Information Irom H arcum Graduates^ 
1968-1977 . Boris alai , Jr , Bryn Ham: , Penn , t Harcum 
Junibr "Co liege, August 1977, 5 pp, (ED 142 270) 



DESCRIPTIVE ANNOTATION f 

The overall purpose of Harcum Junior College's Institu=" 
tional Research Reports is to provide information for "effect" 
ing improvements in eKisting practices." This document is a 
report of their self ^evaluation program, on-going since aca^ 
demic year 1967-^68, designed in consonance with Harvard 
professor David Reisman's statement * * ^ college 

must be tested by its products, the most obvious product 
being the alumni p" Specifically addressed is the eKtent to 
which the College has achieved its objectives " [to] provide 
transfer programs paralleling those offered in the first two 
years of four-year collegiate programs of study," and "[to] 
offer career preparation, equipping individuals to become 
contributing members of the working world*" 

From 1968 through 1977, a questionnaire was mailea each 
summer to that year's graduates* The average respDnse rate 
was 52%, Although the questionnaire was not discussed, this 
report summarizes the data collected over that decade. 

Among the results presented are theses 

• Th^: lowest percentage of those accepted for 
transfer to advanced standing was 30% in 
1975; the highest, 62% in 1973. 

• The lowest percentage receiving full-time 
employment was 201 in 1973; the highest/ 
52% in 1969, 

9 In response to the statement ", , , Harcum 
prepared me for the field I have chosen," 
89% responded "very helpful" or "somewhat 
helpful"; only 11%, "not helpful." 
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graduates into the cOTOiunity at large . . * [evidences] a 
record of effective results J' 

This report was written by the Director of the Harcum 
Junior College Office of Institutional Research. 

EVAIrUATIVE COMMENTS I 

This is a very brief report/ meant for internal use at 
the collage. As such, it provides policy makers and staff 
with a descriptive overview of some outcomes of Harcum 's 
blend of liberal arts and an occupational program. However, 
the report gives the reader insufficient information for 
judging the' credibility of the data. Was the questionnaire 
valid and reliable? What was the extent of the nonresponse 
bias? 

A further problem is that no standards are provided 
against which to compare the reported percentages. How is 
a finding that 11% of respondents found the college to ba 
"not helpful" in preparing them for their chosen fields use- 
ful to policy makers? Is that percentage so small aw, to ba 
not worth considering or so large as to challenge the college 
to initiate changes aimed at improving its occupational 
preparation programs? If it is not usefu7. information ^ ona 
may question the inclusion of the questior^ in an evaluation 
study . 

* * * 



The Effectiveness of Vocational and Technical Pr o grams ^ A 
^" NaTibhal Followup Survey . Gerald G. Somers, in collabor 
ation with~Laure M. Sharp and Thelma Myint and with the 
assistance of Susan Pernbach Meives. Madisoni Center 
for Studies in Vocational and Technical Education ^ Univer 
sity of Wisconsin, 1971, 282 pp* (ED 055 190) 



At the time of the study thcrG were no national data 
available for an evaluation of the effectiveness of vocational 
education; thus the study was undertaken to fill that. void. 

The universe^ the sampling techniques , the study samples , 
and non^response bias are thoroughly described and several 
tables are included. Basically, a stratified random sMpling 
technique was used, resulting in samples of graduates from 
high school vocational programs; graduates from postsecon'- 
dary vocational schools {presumably vocational-'technical in-- 
stitutes) ; graduates from vocational programs of junior col-- 
leges; 6 graduates of high school academic programs; and drop^ 
outs from high schools ^ postsecondary schools, and junior 
colleges , 

Samples of 1966 graduates were surveyed by mail in 1969, 
Questionnaires were mailed to 7327 graduates of 194 secon- 
dary schools, 3461 graduates of 54 postsecondary schools, 
and 2591 graduates of 64 junior colleges. The samples of 
dropouts obtained were too small to include in most of the 
evaluative analyses, but some descriptive data are provided, 

A random sample of non-respondents was drawn and follow- 
up telephone interviev.^s were conducted so non^response bias 
could be assessed. There did not appear to be any consis- 
tent# significant differences; however, it is noted that the 
respondents and sampled non^respondents are not evenly dis- 
tributed among the program areas* 

The development of the mailed questionnaires is not dis- 
cussed; they are a;^pended, however. The questionnaires had 
from 47 to 7 0 questions, with multiple-choice and open-ended 
items. The data analysis consisted of cross^tabulations of 
percentages, chi^squareb^ regression analyses. 

Findings of the study are discussed in six chapters: 

m The Educational Experience and Student Attitudes 

• Post-Vocational Employment Experience 



• Post^Vbcational Educational Experience 

• Evaluation of the Costs and B^efits of 
Vocational Education 

• Conclusions and Policy implications 

Several tables are provided throughout the chapters and 
in an appandiK. Seven "highlights" are provided in the front 
matter of the doGumenti and each chapter has a summary con- 
clusion. 

Conclusions and policy implications, in the form of a 
discussion of the significance of school level, geographic 
region, urban vs. rural setting, program area, demographic 
character istias, and grade point averages as influences on 
attitudes, labor market performance, and further educational 
exparienca are presented in the final chapter. 

Several summary observations are included with their 
impliGations ; 

• Vocational programs at differant school lev^^is 
serve different clientele (becauBe of variances 
in student backgrounds and attitudes) and dif- 
ferent training purposes/ despite similarity in 
occupational program titles. 

m A relatively large proportion of junior college 
agricultural, technical, and health -^^^-^rma 
graduates and high school distribu- 
trade and industry graduates movu from 
graduation to a full-time job or L edu- . 

cation, whereas graduates of other : j-ams 
require longer job-search times. 

• Junior college vocational graduates in general 
eKperience greater economic success with mort-3 
satisfactory employment and better earnings 
than other graduates, both vocational and 
academic . ^ 
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Northeast^ and North Ca/-Tral th?in in 
the South; and high^-i^ 2a^: urban and 
suburban graduates zc^^i Sor rural grad- 
uates * 

• Males ^ older graduates^ and mc^rriert grsd-- 
uates enjoyed labor msr/:Dt experience 
supsrior to those of femaleii^ younger grad-- 
uates and unmarried graduates r one impli = 
Cation is that there is need for a longer 
period of general education preceding specific 
vocational choices , 

m The relation of job to training does not 
appear to influence levels of employnient 
and earnings : actually many students are 
able to earn more by moving out of their 
field of training upon labor-market entry. 

• This may support the view that general voca^ 
tional training and training for clusters of 
job skills are preferable to specific train= 
ing and training for specific jobs, 

m Although occupational program area is of 
some importance in the labor market exp-- 
erience of gr-^duates^ it is not as important 
as traditionally believed. 

This research was funded by the Division of Adult and 
Vocational Research^ U. S. Office of Education^ Department 
of Health/ Education ^ and Welfare. The senior author is a 
professor at the University of Wisconsin; his collaborators 
are staff of the Bureau of Social Science Research # and the 
junior author is a graduate research assistant at the Uni-- 
versity of Wisconsin. 

EVALUATIVE COMMENTS i 

This study is too compleK to permit a thorough critique. 
The authors ackiiowledf^e some limitaftinn^ n£ ¥h^ Ha^s r«n11^r^4- 



graduate of a postsecondary vocational school? What specific 
methods of statistical analysis were used? How was the quas- 
tionnaire developed? 

* * * 



E mployer Satj^sf action with the Skills of Vocational Educat ion 
~ 5radu a. tes in North D a:kota. Larry L. Smiley. Grand Forks; 

Bureau of Educational Research and Services, University 

of North Dakota, August 1976, 81 pp. 



DESCRIPTIVE ANNOTATION I 

One of the goals of North Dakota's State Plan for Voca- 
tional Education specifies the percentage of completers who 
will be placed in the field in which they have been trained, 
or in a related field. This study was done to aid that ef- 
fort by ascertaining how satisfied employers were with the 
completers they had hired and in what areas the vocational 
programs needed to be improved. 

The initial data base comprised all secondary and post- 
secondary sites in North Dakota with at least five vocational 
programs. Personnel in each program identified three major 
employers of program graduates and one or two self-employed 
graduates. The 482 names identified coristituted the survey 
sample. Prom 226 employers indicauing a willingness to par- 
ticipate in a personal intsTrview, 191 provided data. A 
questionnaire (appended to the document) was developed for 
use in the personal interview, which took approximately 30 
minutes. 
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FindingB are presented in the order that questions appear 
on the interview schedule. Responses to each question are 
summarized in tabular form along with a brief discuseion. 
It is left to the reader to draw conclusions. The reported 
findings include the following: 

• Fewer than five per cent of the responding 
employers rated the vocational education 
system below average, 

• 73% of employers rated manual job skills above 
average and 47% ascribed the sama rating to 
practical job knowledge. 

• A ipecific employer commt^;t if. "Most of 
the vocational education ^ i^dumtes come to 
work with skills well dmv^do^ -'d, but the 
training programs have fa.:.lj-d to offer the 
students anything that prepares them for 
moving into supervisory or managerial posi= 
tions. " 

m Nearly 9 0% of the employers rated the voca-- 
tionally trained employee high to average 
in comparison with those without the voca-- 
tional training. Many could not make a Gom= 
parison because certification requirements 
for entry-level job placement prevented a 
non-^vocationally trained individual from 
being hired . , 

m Several "needed areas of improvement for 
job applicants" were listed. Heading the 
list (in order of frequency of response) were 
attitudes toward work and appearance. Also 
listed were courses that vocationally trained 
students should have/ with courses in Communis 
cation^ Speaking Effectively^ and Work Orien- " 
tation at the head of the list. Need for a 
course in Success in Marriage was listed by 
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the graduates is provided, ^i-^^n^ the conmients 
arei "Realitiei of position nee.i to be made 
clear to the students,'* and "Secretaries trained 
as medical secretaries are less capable than 
those with no specialized training." 

Several employers expressed the opinion that 
high school graduates are sold a bill of goods 
when told they have a saleable sXill after their 
high school vocational training i that the intent 
of high school vocational training im to introduce 
young people to vocational areas and materials 
of those trades, not in any sense to provide a 
terminal degree. One employer said/ "If this 
high school level training has helped a young 
person to clarify vocational interest which 
can be furthered by additional schooling ^ it 
has served an excellent purpose*" 



EVALUATIVE COMM^iMTS ^ 

This type of study can alert vocatioaal planners to ^ 
employers^ perceptions of vocational education, thus Indxcat 
ing areas where the system is meeting their needs, where 
improvements are needed, and what kinds of changes may prove 
beneficial, it gives direction for more intensive investi- 
gation, h careful reading of the report is necessary, how- 
ever, because in seme instances the author notes that re- 
spondents' verbal conmients contradict the ratings. For ex- 
ample, employers' ratings of students' attitudes toward work 
do not indicate that this in a serious problem, wher^iias em- 
ployers ' cominents indicate that it is * 

It is unfortunate that the findinge are not broken down 
by level of schooling (secondary vs , postsecondary ) and that 
information about the content of the various vocationa^ cur- 
ricula was not compiled. Although thie study was not intend 
ed to be generalized to programs outside North Dakota,, such 
information would be helpful to those who might wish to ^ use 



h-57 



"A Follow-^up Study of Fomer Occupational-Technical Students 
at Virginia Conimunity Colleges." Edith H* Carter* Paner 
presented at the Annual Regional Research Conference for 
Secondary and Post'-Secondary vocational Education Person-- 
nel, April 17, 1976, 10 pp. (ED 136 899) 



DESCRIPTIVE ANNOTATION I 

Xt is stated in this report "The planning for axpanaion 
of program operation should be based upon awareness of in-- 
formation about students and potential students, their 
occupational needs ^ and also the activities^ of former stu=- 
dents after they leave the community college^" This study 
was designed and conducted to provide comprehensive and ac- 
curate information about former students of Virginia conunun-- 
ity college occupational=technical programs^ Explicit, major 
objectives of the study were as follows: 

• To identify selected personal and demographic 
characteristics of former students jn occupa- 
tional-technical programs 

^ To identify post-college activitlrf^ of former 
students 

^ To study the attitudes of former Btvidarits 
toward their community college experience 
and current employment 

^ To stuu 7 patterns of student retention and 
withdrawal 

To eKaiaine differences among graduatea dnd 
non-graduates and among the several types of 
graduates in terms of their characteristics, 
post^-college activities, and personal evalua- 
tions of college experience and employment 



The fA^pulation consisted of all students who had been 
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about post-collrie activities^ employment , and avaluation 
of college m^B .^^aes, The student questionnaire was de^ 
stgnad to fulfill the stated objectives of the study, but 
no further discuesion of its development is provided, nor 
is it appended. 

Responses to the questionnaire were entered onto computer 
tapes by optical scanning to obviate the possibility of key- 
punch errors* No statistical analyses of the data are re-^ 
ported • 

Two results reported are (1) that almost three-fourths ^ 
of the [respondents 1 were working full-time in jobs related 
to their Gonmunity college curriaula and (2) that both grad- 
uates and non-graduateB rated tt-;^ rnality of their college 
preparation superior or good in nout areas with nine-^tenths 
of th^ stating that the-^' ^-^uid leoommend their cominunity 
college to an individual . inin^ to »::^mroll in the same pro- 
gram - 

This study was conducted/reported by the Director of the 
Office of Institutional Research^ New River Community College* 



EVALUATIVE COMMENTS i 

This report is necessarily brief, because it wa^ a con-- 
ference paper rather than a final report of a study. Still, 
the crf-dibility would be enhanced if more complete analyses 
of the data were presented. The relatively large response 
rate reported may be indicative of. the advantage of follow- 
up mailings , 

As in most of the studies annotated in this bibliography^ 
there is some question regarding the validity of the survey 
instrument. For eKamplei what assurance is there that former 
students who respond to a questionnaire from their alma mater , 
when asked if they would recommend it to other individuals, 
will respond honestly after thoughtful consideration? A 
tdlot-test in which respondents are questioned more thoroughly 



ing vocational eciucatj.an legislation and expenditures occurred 
in the absence of adequate inf orination , " He goes on to 
say that althQUgh research has been done and some of it is 
encouraging in its diversity of perspectives ^ the policy rele-- 
vance of the research "has been circumscribed by limitations 
inherent ih the design of many of the studies, or as a result 
of conoeptuai or methodological dif £ iculties , often compound^ 
ed~--if not caused-=by a lack of adequate data." Other limi-- 
tations are discussed as well. 

This study was conducted to contribute to "improvement 
in the formulation of public policy on the education and 
training of youth , , * by reviewing completed research and 
by extending and refining it with both improved data and 
methods or analysis, ■* The empirical aspect of the study seeks 
to determine how youth from different high school programs 
compare in the following areas i 

• knowladga of occupations 




L survey conducted as part ot thm Nationai 
/eye. The in.atrumantation is "described in 
nents * All analyses in the study are based 
::ally weighted to allow reliable comparisons 
i white youth. Further delimitation of the 
3S of vocational program studies or extent of 
Dssible; however^ it was possible to statis-- 
:or the following demographic variables s 
itus^ BCholastic aptitude^ and race. Other 
i used are price level and demand^ military 
^e at age 14 (rural farm, rural non--farm, 
m city^ and large city or suburb) r and, for 
ith analyses, grade placement and work eKperi 
All the variables used, independent and de-- 
roughly described . 



5end:.:i^. of Key Related Literature for informa^ 
itional Longitudinal Surveys. 
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• The lowest percentage receiving full-tinie 
employment was 201 in 1973; the highest/ 
52% in 1969, 

m In response to the statement " , , . Harcum 
prepared me for the field I have chosen^" 
89% responded "very helpful" or "somewhat 
helpful*'; only 11%, "not helpful." 

Among the interpretive comments made by the author are^ 
ne reason for the decreasing trend in transfer to four^ 
ar colleges/universities TOMy be that m.ore Harcum graduates 
e entering oeoupational fields that do not require bacca- 
□reate preparation^" anO "Some graduates do not look for 
rk until the fall following graduation, and therefore the 
rcentages reported are conservative estimates of the employ 
nt status of Harcum graduates." The general conclusion 
this decade report is "the articulation of its 
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graduation to a full-time ; 
catiorM whereas graduates ( 
require longer job-search ' 

m Junior college vocational c 
experience greater economii 
aatisfactory employrnent an( 
than other graduates, both 
academic * ^ 

# With the exception of agric 
euonomic benefits of vocat: 
education are higher for gi 



The authors acknowledge that 
programs are cost-ef £ eotive; that 
from other studies; and that juni< 
not necessarily the best choice f( 
of variance in interest^ intellig^ 
tions . 
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Vocational Reseirch/ U, S. Office of Education/ Depirtnient 
of Healthy Education ^ and '^elfara, The senior author is i 
profiBsor at the University of Wisconsin; his colliborators ' 
Uri staff of the Bureau of Social Science Risearch/ and the 
junior author is a graduate research asiistint at the Uni- 
versity of Ifisconiini 

IVaLUMIVE COMMENTS: 

This studv is too conipliK to periit a thorough critique. 
The authors acknowledge sonie liniitations of tha data collect- 
id and the data availabli^ as will as noting that/ in man^ 
insti:ices/ data are unavailabli to distinguish among altirnitivi 
riasons for some findings* For Bm^Ui did women as a 
whole iirn lass bicausi of overt discrimination or bicausi 
^thiy did not aspire to full-timi jobs following thiir voca- 
tional triining? 

Although the report ii lengthy^ some information neces- 
sary for assissing tha study and relating its findings to 
other studies is lacking* For example^ whit exactly is i 



iplopriofpfiiiaHi 
pits, iiffliisiliffieirt^^^ 

illpiiliitw 



liiWii ilii trt ippiiitel! J 



its. inf tlM*Pi ii nitip^^^ 
of iipiiif ttili 



ifil "iJi ill i iifiiil k 



list 111 sir i fiiif of i^m m 
ittltiJi tiii ii ill iinii ^ 
M m mE tilt iiiioiilf tali 
iiiliiiifiiiBi^ 



iti, liiiiif liili 
tMi it tli lii i U li^ti U k i 
IB ifiSii littpilteiif 



i If iflofff! itiot Ml ts tsilt 
plliii Iflofi, li liMiif it-sl- 
01 M III iikrlf itirti tin 
' ill dsi toe mi i 



tisi to iiiti'iool iM k 



1 1 ll^t if m i U filil Wisiti iplp 
lii M fitisii ilsitii itii^ ail 



spondents' verbal comments aontradict the ratings. For ex- 
ample, employers' ratings of students' attitudes toward work 
do not indicate that this in a serious problem, wheri.'as em- 
ployers ' comments indicate that it is . 

It is unfortunate that the findings are not broken down 
by level o£ schooling (secondary vs. pos t secondary ) an<3 that 
information about the content of the various vocational cur- 
ricula was not compiled. Although this study was not intend 
ed to be generalized to programs outside North Dakota,, such 
information would be helpful to those who might wish to _ use 
the study to suggest areas for further explcration withm 
their own systems. 
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A Follow-up Study of Illinois Home Economics Job Training 
Programs.' Final Repo rt, Kathleen M. "Howell and Joyoo 
L. Pelstehausen. Springf iold • state of Illinois/ Board 
of Vocational Education and Rehabilitation, Division of 
Vocational and Technical Education^ Research and Develop- 
ment Unit, September 1971, 122 pp. (ED 059 379) 



DESCRIPTIVE ANNOTATION i 

This pilot study was designed to test the use of certain 
procedures and instruments for examining the post-'high achool 
employment eKperienceB of graduatee of Illinois Home Econom- 
ics Cooperative Education ProgramB and to provide recommenda-- 
tions for leaders in the field. Both graduates and employers 
were surveyed. Questions asked on the graduate questionnaire 
concerned type and number of jobs held/ reasons for leaving 
jobS/ desirable and undesirable features of jobS/ part-^ or 
full-time status, reasons for never being employed, and the 
value of the job training received. The einployer/Bupervisor 
questionnaire asked how well the graduate was prepared for 
the job/ how proficient the graduate was in important job-- 
entry skills, and how the job training for that position 
could be improved. Both gueattonnaires included multiple- 
choice and open-ended questions. 

Cooperative Gainful Home Eaonomics teachers in 44 Illi'- 
nols public secondary schools identified 691 home economics 
graduates of 1969 and 1970. The sample analyzed consiated 
of 188 respondents/ a responBe rate of 27% (with non-deliv-- 
erablei assumed to be random, a response rate of 34%) . 
Those graduates identified 110 employers of whom 75 returned 
questionnaires, yielding a response rate of 68%* 

Both the graduate and the employer questionnaires were 
adapted from others developed and used previously by the 
senior author. After pilot testing for use in this study, 
revisions were made* The questionnaires are appended. 

The results were tabulated according to seK/ graduation 
year, and courses taken. Tables of frequencies/ averages, 
and percentages are provided, but no further statistical 
analyses were made* 

The following findings from the graduate questionnaire 
were included i 

• 62% of the graduate sample were employed at 
the time of the study. 
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• 24% to 551 of those employed (varying 
with training area) were employed in 
their areas of training . 

• 11% of the graduates had never been 
employed . 

• 51% of the graduates indiaated job sat- 
isfaction related to working with people, 
whereas 54% indicated dissatisfaction related 
to job-situation factors, such as specific 
undesirable tasks and working conditions. 

• Graduates felt that, aL though they would 
have liked more employment experiences and 
instruction, their vocational programs had 
prepared them for employment, especially m 
getting along with others, using time and 
energy, and handling new or unpleasant sit- 
uations. 

Following are some of the findings from the employer/ 
supervisor questionnaire reported: 

• Job training contributed to employability 
traits, especially in areas of attendance, 
attitudes toward supervision, and coopera- 
t iveness , 

• Personal factors such as initiative and 
honesty were entry-level skills considered 
important . 

• Training should emphasize positive attitudes 
toward work, learning, and supervision. 

Specific recommendations are made for the Program Appro- 
val and Evaluation Unit, the Special Programs Unit, and the 
Research and Development Unit. These range from recommenda- 
tions that specific steps to Implement the provisions of the 
State mil be identified to suggestions that a comprehensive 
evaluation program include long-range investigations to de- 
termine relationships between crime and delinquency rates 
and job training (or lack thereof) for youth. 

Both authors are from the Eastern Illinois University's 
school of Home Economics. The senior author, project direc- 
tor, is an Associate ^ofessor whereas the Dunior author, 
orincipal investigator, is a research assistant. The study 
Sas coSdioted ?uisuant to a contract with the Illinois Board 
of vocational Education and Rehabilitation. 
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EVALUATIVE C0MI4ENTS i 

The authors themselves question the validity of their 
results, since only 34% of the graduates who received ques- 
tionnaires responded. They express concern that 3 5% of the 
mailed questionnaires were undeliverable by the posit office ^ 
indicating lack of current address inforxnation even though 
follow-up is mandated* 

One of the notable features of this study is its pilot 
test and appended questionnaire. Reliability and validity 
data are not given but it is noted that some item analysis 
has been done, indicating that the instruments will be re- 
fined for still further use- 
it * * 



Follow-up Study of Machine Tool TechnolQgy and Bu ilding Con- 
struction" Graduates: " Evaluation Report ; Wilbur Ha31, 
Rodney Gray, and" Arthur O- " Berry / South Portlandi South- 
ern Maine Vocational Technical Institute, August 1975, 
115 pp* {ED 124 746) 



DESCRIPTIVE ANNOTATION; 

This study was designed to determine how well the South^ 
ern Maine Vocational Technical Institute's Machine Tool Tech-- 
nology (MTT) and Building Construction (BC) prograiTis are suc- 
ceeding in their efforts to prepare well^qualif ied employees. 
Specifically, it sought to acquire the following information: 
(1) employment status of graduates, (2) employment sequence 
of graduates, (3) perceptions of the value of the MTT and BC 
programs held by graduates and employers, (4) up-grading and 
retraining needs, and (5) graduate and eriployer profiles for 
prograinmatic restructuring and development. 

h questionnaire was sent to each of the 126 MTT and BC 
students from the classes of 197 0, 1972, and 1974. A random, 
sample of those graduates living within a reasonable coinmut- 
ing distance of Southern Maine Vocational Technical Insti-- 
tute were selected for personal interviews to supplement the 
questionnaire. Seventy^five graduates returned questionnaires 
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and 38 of them were interviewed. Employers were interviewed 
when possible. The response rate was 63% for BC graduates 
and 551 for MTT graduates. A majority of the sample had re- 
ceivea diplomas? the others, associate degrees. 

The questionnaire, consisting of an "opinionnaire" and a 
checklist, was validated by use v^ith a sample of graduates 
before being sent to the selected sample. Although personal 
information (name and address) was collected, graduates were 
assured that their responses would be kept confidential. The 
opinionnaire and checklist are appended. 

The document provides extensive tables detailing re- 
sponses to the survey instruments. Responses to rtiany of the 
open-ended questions are listed as well. Results, in terms 
of percentages, are reported separately for MTT graduates and 
BC graduates, although no statistical analysis is provided. 

Among twenty summary findings are the following: 

• All graduates who respoiidsd were employed, 
with a majority working in the trade area 
for which they had been prepared or in one 
closely related to it, 

• The majority had held only one job since 
graduation . 

• 38.6% of the BC graduates and 65.7% of the 
MTT graduates had pursued further training 
or education. 

« Respondents indicated general satisfaction 
with the quality of the instruction they had 
received, although attaching many suggestio'ns 
for ittiprovement in the curricula. 

• Employers were generally satisfied with program 
graduates, although they ^ere concerned with 
continued program relevancy to the trade or 
occupational areas." Eecoitmendations for pro- 
gram and study improvenient are provided. 

This study was initiated by the Southern Maine Vocational 
Technical institute. The principal investigators were in- 
st?uStSrs at^the instituta-^one BC, the other MTT--employed 
full time for two months to conduct the study. Dr. Arthur O. 
Berry, whose professional identity is not reported, was re- 
sponsible for the data analysis and the evaluation report. 
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EVALUATIVE COMMENTS i 

The report gives sufticient information for replicating 
the procedurea of the Btudy. However, other information is 
laakingi The interview questions for the graduates and the 
employars are not deecribedF although it is stated that the 
guastionnaire was validated, the validation procedure and 
resultB are not described; although the 100% employment rate 
of the respondents suggests a tendency for bias toward the 
employed, possible bias in the respondents is not considered; 
and finally, statistical analysis of the data was not done- 

Although these insufficiencies may have minor impact for 
the intended audience'-'-Southern Maine may conclude that their 
Vocational Technical Institute is doing a reasonable job of 
preparing well--qualif led employees^ — ^it is injudicious for an 
expanded audience to draw any conclusions from this report/ 
or even to determine whether the survey Instrument would be 
useful for another institution. Without even minimal sta- 
tistical dat^ analysis to determine the probability of random 
differences in responses, on the basis of this survey it is 
risky even for Southern Maine to draw any iubstantive con-- 
elusions about policy or program changes needed. 



Follow-up Study of 1969-1975 Graduates of the Division of 
Technology of New York City Conm\unity Colleges . Jeanne 
M, Gammel, Stanley MTBroasky, and Richard Alfred. 
Brooklyn: New York City Community College, April 197 6, 
134 pp. (ED 121 396) 



DESCRrPTIVE ANNOTATION i 

The preface to this study states, "If the college off the 
future is to realize even a modicum of its potential, ff acuity 
and administrators will have to begin to pay as much attention 
to the outputs of instructional programs as to the number of 
students, buildings, and organizational structures,*- The 
study was designed to provide such output Information In terms 
of career patterns, perceptions of curricula, and attitudes 
toward work and further education held by graduates of twelve 
curricula of the Division of Engineering Technology at Mew 



91 



A-64 



York City Community College mccc) . The results were in- 
tended to be used not only internally but also externally 
to provide agencies with informatian regarding contributions 
made by the college to business and industiry. 

The 2,554 students graduating over the seven-year period 
£rom 1969 through 197 5 constituted the population. In the 
Call of 1975, a questionnaire was mailed to 2,087 of those 
graduates, though the method of selecting the sample is not 
stated (perhaps mailing addresses were not available for the 
test of the population). Nine hundred twenty-two graduates 
returned usable responses, yielding an overall response rate 
of 44%. A table lists the number of graduatcso by curriculum 
and graduation year and also indicates respective response 
rates . 

The development of the questionnaire is not discussed, 
nor is reliability and validity information provided; however, 
the instrument is appended. The questions asked were designed 
to collect data on the following variables s (1) present occu- 
pational status; (2) field retention of employed graduates; 
(3) proportions in senior, supervisory, managerial, or pro- 
fessional positions; (4) chang© of employers and job titles 
between first and present jobs; (5) further college atten- 
dance; (6) further degrees sought or Gonipleted; (7) percep- 
tions of the NYCCC curricula; and (8) other variables related 
to employment and further education, 

rhe resulting data for each dependent variable are pre- 
sented in two-dimensional frequency/percentage tables of 
graduation year by curriculum. Sunimary results from four- 
teen earlier follow-up studies are presented for comparison. 
NO statistical analyses were done. The authors point out 
that results of this study cannot be considered positive or 
negative because standards for success have not been estab- 
llilied. For example,, no one has determined how many students 
should be employed in training-related fields or what per- 
centage of students should continue their education. 

The results of the study are discussed in detail and hy- 
potheses are suggested to explain differences in graduates 
of various years and curricula. Among the results discussed, 
the following are included: 

• 77% of the graduates are employed, 14% are 
full-time students, 9% are unemployed, and 
1% are in the military service or retired. 

• Of the unemployed 1974 and 1975 graduates, 
most are students, whereas few of the un- 
employed 1969-1971 graduates are students. 
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ft The proportion of employed grtiduates by 
curriculum area varies from 4 7% for arch- 
itectural teGhnology to 96% for fire- 
protection technology. 

m Of the 607 graduates employed in training- 
related jobs^ 44% have senior, supervisory, 
managerial, or professional positions. 

• Of all employed graduata respondents, 27% 
found their education eKtremely helpful in 
performing their present job, 38% fairly 
helpful, 24% slightly helpful, and 11% not 
at all helpful. 

m 61% of the respondents attended college 
after graduation from NYCCC , ranging from 
58% for the class of 1974 to 68% for the 
class of 1971, To summarize the findings, 
graduates of each curriculuin are described 
in "modal student profiles 

Although the study was clearly done internally, both by 
and for Qollege ataff, the researchers were not identified. 



EVALUATIVE COMMENTS^ 

This study was well thought out and is one of the few 
in which the particular data collected can be traced to an 
overall purpose. It is quite possible that one fault is an 
omission" in reporting rather than in the study itaelfi A1-- 
though information from fourteen previous studies is pre- 
sented, there is no Indication whether the same instrument 
was used- The point is twofold^ Are the data comparable 
and has the instrument been tested , 

Many comparisons are made among graduates of varying cur- 
ricula and graduation years. Although hypotheses were de- 
veloped to explain some of the dlfferenceB, no statistical 
analyses were performed to indicate whether they were likely 
to be merely chance differences. Thus., the need for further 
research designed to test those hypotheses is indicated. 
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"A Follow-up Survey of the Graduates of the High Schools %n 
vocational Region #8, New Hampshire." Roger D. Crim and 
Eugene W. Ross. Paper presented to the New England Edu- 
cational Research Organization Annual Meeting, Province- 
town, Mass., May 6, 1976, 32 pp. (ED 123 471) 

DESCRIPTIVE ANNOTATION: 

This study, conducted to obtain a data base for New 
Hampshire vocational Region #8 public secondary vocational 
educators, is intended to help them plan future curricula 
and programs. The issues addressed were education and train- 
ing beyond high school, employment status during the first 
year after graduation, and current employment status. 

The population comprised the 1968, 1970, 197 2, and 1974 
graduates of the six public high schools within Region #8. 
?he population was stratified by school and graduating class, 
and a random sample was taken from each of the 24 strata. 
The sample (1256) was 50.5% of the population (2509). 

An interview form was developed, using the Dictionary of 
Occu pational Titles as a guide for categorizing vocations . 
The Interview schedule is appended. Telephone interviewing 
was conducted by paid, trained interviewers over a two-week 
period. When the high school graduate could not be reached,, 
information was collected, when possible, from a parent, _ 
grandparent, guardian, or spouse. Information regarding 840 
graduates was collected, yielding a response rate o| 66.4%. 
The response rates for all strata are tabled. 

The data were tabulated in frequencies and percentages, 
and a composite profile for the region was obtained. No 
further statistical analysis was done. 

Among the 36 findings listed are the following: 

• 48% of the graduates received no additional 
education or training. 

• 80% of the graduates were employed immediately 
after graduation. 

• At the time of the study, 63% of the graduates 
were employed full-time, 

• 30% of those employed were employed in clerical 
and sales occupations, 21% in service occupations. 
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The authors conclude that all six of their major voca- 
tional education areas are represented in the employment 
profile and that although 261 of the graduates indicated 
they were unemployed, only 7% were looking for work. Spe-- 
ciflc recorranendationB for the Laconia Area Vocational Center 
are made, 

This study was sponsored by the Department of Health, 

Education, and Welfare. The senior author is an Assistant 
Profleisor of Education at Plymouth State College^ the junior 
author is the Vocational Education Director of the Laconia 
Area Vocational Education Center #8. 

EVALUATIVE COMMENTS s 

Prom the stated purpose of the study, the reader would 
assume that the population consisted of graduates of voca-^ 
tional programs; however, this fact is not eKpllcitly stated. 
Further, no differentiation in the various types of vocational 
courBes taken by students was made, As a consequence, it is 
difficult to account for the reported employment rates, 

' Relying on sources other than graduates themselves to 
determine Miployi^ent status posed a serious methodological 
problem. Further, there is no indication of how much data 
were collected from secondary sources* without some sort of 
verification, the data cannot be considered reliable. 

Since data were collected from graduates of several years, 
it is unfortunate that some form of trend analysis was not 
done which would be particularly useful if it could be ref- 
lated to any curricular changes made in those years, 

* * * 
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The Impa ct of Secondary School Occ upational Education in 
MaBsachuietts . ~ William G. Conroy , Jr., and Daniel E. 
fiiaitiond. Lowell: University of Lowell, College of Man 
agement Scienee, Spring 1976, 43 pp. (ED 122 095) 



DESCRIPTIVE ANNOTATION! 

According to published goal statements, Massachusetts is 
committed to, provide youth with the skills and experiences 
necessary for entry and success in the labor market. Because 
of this commitment and because of the wide variety of occu- 
pational education programs offered in Massachusetts, this 
study was conducted to examine the impact of high school pro- 
grams on the life style of students: college preparatory, 
general/academic, and occupational programs. It was intend- 
ed to provide a knowledge base for improving secondary edu- 
cation policy. 

The study focused on school/labor-market relations but 
also addressed citizenship and leisure-time activities. The 
conclusions drawn and recommendations made in this report 
are not based on an exhaustive analysis of the data collected. 
However, the data are available from the Massachusetts Divi- 
sion of Occupational Education for anyone who may wish to 
explore the data base further. 

The research design was ex post facto: A sample of stu- 
dents who graduated (or "should have") from Massachusetts high 
schools in 1969 and 1973 were asked to respond to a question- 
naire mailed in the summer and fall of 197 5. A multi-stage 
sampling plan was used: first schools, then students, A _ 
figure depicting the variables considered in the total design 
is provided. Questionnaires were mailed to 7,894 students; 
2642 responded, 3452 did not; and there were 1800 non-deliver- 
ables. If non-deliverables are assumed to be random, the 
response rate was 43%. Although the findings are not cor- 
rected for bias, the data are weighted according to school 
sampling characteristics. 

The survey instrument is not discussed or appended; how- 
ever, categories of the questions asked are included in the 
figure depicting the study design. There is some discussion 
of the operationalization measurement of the variables; vot- 
ing behavior is used as a "proxy" for citizenship? salary 
data are presented in terms of purchasing power of 1975 dollars 

Fisher t-tests, chi-squares, and correlation coefficients 
are used to analyze the data and their uses are _ described . 
The data collected about major non-educational influences on 
lifestyle were used as controls so the independent affects of 
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school programs could be estimated. TablGs breaking the data 
down according to such variables as school program, grade 
point average^ and postsecondary education status are pre-- 
sented. Among the eleven summary findings presented and dis-= 
Qusied are these i 

m There were no differences in average income . 
batv/een females who had occupational training 
and those who had none, 

m For Btudenta who did not go on to postsecondary 
education, the average annual salary of 1973 
occupational students was higher than that of 
nori--oCGUpational students (a finding extremely 
pronounced for male Trade and Industrial atu^ 
dents) . 

m Two-thirds of occupational students did not 
work: in fields for which they had trained* 

Among the conclusions discussed are theses 

m The superior labors-market behavior of occu= 
pational students seems to result from the 
school program as an independent entity ^ 
because the effect eKists^ even after account- 
ing for variables such as socioeconomic status 
and scholastic aptitude* 

• Secondary schools play a minor role in help- 
ing youth find employment. 

Six recommendations for Massachusetts policy makers are pre- 
sented. 

This study was initiated by the Division of Occupational 
Education^ Massachusetts Department of Education. It was 
conducted by professors in the College of Management Science/ 
University of Lowell* 

EVALUATIVE COWffiNTS i 

This study is notable for the completeness of the infor^- 
mation given^ although it is unfortunate that no information 
regarding the survey instrument is provided. The design is 
clearly laid out, data were collected on intervening vari'- 
ables, and response bias was considered^ although not analy- 
zed. Several aspects of the questionnaire can be inferred 
from the discussions of suimTiary findings but there is no 
indication of its reliability or validity* 
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Another notable aspsct of this report is that of offer- 
ing readers an opportunity to further explore the data base. 
Not only does this allow analysis of data collected but not 
analyzed, but also it allov/s analyzed data to be reanalyzed, 
perhaps by use of different methods. 



* * * 



"Intensity of Occupational Training: Its Effects on Subse- 
quent Lbor-Market Experiences." Robert E. Allen and Thomas 
G. Gutter idge. Community/ Junior College Rese arch Quarterly 
vol. 2, pp. 367-380, 1978. 



DESCRIPTIVE ANNOTATION: 

This study was designed to investigate the relative ef- 
fectiveness of occupational programs of public and poP^J?^ 
tary schools at the postsecondary level. Although both kinds 
of institutions are designed to develop job skills, they tend 
to differ in the proportion of job skills training and liberal 
arts education offered. Two other studies have investigated 
this issueS but neither controlled for two important vari- 
ables likely to affect postgraduate labor-market activity; 
geographic location and program quality. This study peks 
to provide information which will enable the student to choose 
the tvpe of program suitable to his/her specific abilities, 
needs , and objectives. The data collected to provide this 
information concerned student career aspirations and work 
educational experiences before, during, and after two-year 
college attendance. 



8s M. Jung, V. N. Campbell, and J. M. Wolman, "A Compar- 
ative Study of Proprietary and Nonproprietary Vocational , 
Training Giaduates." Journal of Vocational Behav^g£. 1976, 
8 209-225. Wellford W. Wilms, Public and Proprieta ry Voca 
ti onal Training . Berkeley: CaHf.: Center tor Research 
and Development in Higher Education, 1975. 
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The population comprised alumni who had graduated from 
four upstate New York two-year postsecondary institutions-^'" 
two community colleges and two proprietary buslneBS insti- 
tutes. Because both proprietary schools had been in exist'- 
ence for over 100 years and becauBe proprietary schools are 
sub j act to eKtensive regulation in New York, the proprietary 
and public schools were assumed to be relatively homogeneous 
with respect to quality. 

The sample consisted of a random sample of 50 graduates 
from each school having a graduating class between 19 62 
and 1970, thereby excluding the years 1965, 19 67, and 1969* 

Questionnaires were mailed to 1773 alumni and usable re^ 
sponsee were obtained from 683 graduates (38,5%), but the 
ftl:mb^_was reduced to make the sample more homogeneous by 
excluding non-business majors and graduates who had received 
baccalaureate degrees. The study sample, then, consisted of 
173 proprietary graduates and 105 community-college graduates, 
A £ive--minute telephone interview was conducted with a sam- ^ 
pie of the non-respondents to assess non-response bias . Chi- 
square analyses indicated that respondents and non--respondents 
were not significantly different (.05 level) with respect to 
several variables. 

An eight-page questionnaire was used for this study. Its 
development is not discussed, nor is it appended. One ques-- 
tion asked respondents to indicate for each £ull=-time job 
since graduation (1) the name and address of the employer; 
(2) the scope of the business operation; and (3) the job 
title, functional area and annual compensation of the employ- 
ea. Another question asked to what extent first and present 
jobs were related to training. Job satisfaction was measured 
through the Job Description IndeK developed by Smith, Kendall, 
and Hulin . ^ 

The labor-market experiences of the public and propria^ 
tary samples were eKamined at two time points i at initial 
entry into the labor market and at the time of the survey, 
Chi-squares, t-- tests, and multiple regression were used to 
analyze different aspects of the data. 



^P. K. Smith, L. M. Kendall, and C. C. Hulin, The Measure - 
ment of Satisfaction in Work and Retirement . (Chicago i Rand 
McNally, 1969) . _— _ — — 
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■ Among the results discussed are these: 

• The type of institution attended did not 
greatly affect the measured non-econoinic 
aspects of the later labor-market experi- 
ences. 

• There was a sizable, but not statistically 
significant (at the .05 level) difference 
between present salaries of alumnr of the 
two types of institutions, with those of 
Gommunity college graduates higher. 

• over 50% of the gri^uates were employed, 
both initially and at the time of the 
survey, in poaitions slightly or wholly 
unrelated to their training. 

m conclusion, the authors discuss some of the limita- 
tions of this exploratory study. They aclcnowledge that it 
sSeLei investment rather than ^^^^^^^V^/^^^l^^^^ 
tion that the time frame was short compared to the life 
long'r^ifications of education, and that the study was ^ 
llmftirto four schools In upstate New York. Although they 
it^almt the need for further research, they conclude that 

llenL'cIn"lpplrently, -select ^-^ool with a pedagogy 
consistent with their educational objectives without Deopar 
diaing subsequent career opportunities. 

The researchers are from the Department of Business Admin- 
istrl^ion university of Wyoming, and the ^p^f^nt of organ- 
tlation and Human Resources, State University of New York at 
tf.-Ff^in The research was conducted pursuant to a grant 
!no! 91-3 6-7141) from the Employment and Training Adminis= 
tration, U.S. Department of Labor. 



EVALUATIVE COMMENTS: 

Although this study has limitations, including those 
acknow£edgld by the authors, it is notable for at least two 
™2nnt. (1) It builds on previous research, thus adding 
an iAteirated bSy of knlwledge and (2) An effort was . 
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This conclusion is unwarranted on the basis of a single study 
because it is essantially an acceptance of the null hypothe- 
sis. 1° 

* * * 



Job Selection Patterns i Linkage Brntwemn Vocational Education 
Programs and the Labor Market . George H\ Copa and Bruce 
h\ Klaven , Minneapolis % Minnesota Research Coordinating 
Unit for Vocationai Education^ Department of Vocational 
and Technical Education^ University of Minnesota^ Febru-' 
ary 1977, 161 pp. (ED 138 809) 



DESCRIPTIVE ANNOTATION: 

The purpose of this study was to develop and test a con= 
cept for defining the linkage between vocational education 
programs and the labor market. This concept , termed "job 
selection pattern," is defined as "a description of the jobs 
selected by program leavers in terms of frequency^ occupa- 
tion, industry, geographic location, and tiine, " The authors 
state that meaningful planning of vocational education pro- 
grams (i,e,, planning that will insure gainful employment 
opportunities for prograin leavers) can be done on the basis 
of information concerning labor-market demand only if job 
selection patterns are identified and if the patterns are 
stable from year to year (or instability is e>cplained) , Sta- 
bility of job selection patterns refers to the degree to ^ 
which graduates of a particular program select jobs in simi= 
lar occupations and industries in the same proportions from 



^°See one of the research teKts listed in the technical 
notes section of this bibliography for a discussion of this 
point. Very briefly, if research discovers a difference be- 
tween two groups, it is clear that a difference exists, al- 
though the cause may be questionable. If no difference is 
discovered, however, it may be because the observational tech 
niques and statistical analyses were not powerful enough or 
were otherwise unsuitable for detecting a difference. 
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year to year. In order to test the job selection pattern 
Concept Sn graduates of Minnesota area vocational technical 
institutes, the following objectives were accoinplished i 

■ • Describing the pattern of jobs selected by 
graduates 

• Assessing the stability of the job selection 
pattern of graduates from year to year 

• Describing the job selection of graduates 
by geographic location 

• Assessing the stability of jobs selected by 
graduates from year to year 

• Assessing the longitudinal stability of the 
job selection pattern by graduates 

The information needed to complete this study had been 
collected through the Minnesota Vocational Follow-up System 
(a copy of a portion of the instrument is aPP^f ' 
ever, ill the information for every Program had not been 
coded or stored in the computer system. Therefore, a pur- 
posive sample of program areas with stored information was 
lelectel The critelia used for sele"ting the programs are 
discussed The programs selected were as follows. agri^ 
ticSnoloS;; automotive; clerical training; food preparation 
(cS li ooSks); electronics, machine ^^opr practical nu^^^ 
ing; and sales and business management. Samples of students 
griduating in 1970, 1971, and 1972 were ^^osen to restrict 
Ihe sample size to 100-150 graduates of each program who 
were employed each year. Thus, for some ^^^^""^^ '^J^^ 
sSlI was taken, whereas for smaller programs, all working 
graduates were included. 

Graduates of 28 of Minnesota's 34 ^^ea vocational tech- 

.nioal institutes were included in the ^f^Pl«- f^fSled 
characteristics of the institutes and?he students included 
in the sample are discussed, and a table is Provided indicati 
iumbe?s and percentages of students of those returning the 
foUow'up questionnaire, and of those currently employed 
The averlgS return rate of the Minnesota Follow-up System 
ovL all Jostsecondary area vocational-technical institute 
graduates was approximately 80%. 

Most of the data analysis was accomplished through com- 
piLat?ln of frequency distributions and determination of 
lercentages; chi=squares were used to "sesa the statisti 
cal significance of change in patterns over time. 
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Results of the study, relative to each of the objectives 
lieted in the first paragraph of this annotation/ are dis- 
cuBsed by occupational program area* For each program area, 
a matrix is provided indicating percentages of graduates ein- 
ployed in various occupations (e.g*/ prof essional , sales ^ 
blue-collar) in various industries (e.g agriculture, manu=- 
facturing, finance^ entertainment) . One of the findings 
diBCUssed is the percentage of graduates over the three-year 
period remaining in the same major occupational and industry 
group from the first job to job-after-one-year (from 56 to 
59% for most program areas) , 

hmonq the seven conclusions and recommendations listed 
and discussed are thesei 

• There is a characteristic pattern of jobs 
selected by graduates of vocational education 
programs unique to each program and with 
overlap betv^reen program areas, 

9 The job selection pattern changes during the 
first year after graduates leave vocational 
education programs ^ and the degree of change 
varies between programs > 

m Follow-up data on vocational graduates should 
include the information necessary to construct 
job selection patterns ^ in order that the 
utility of the concept may he further assessed. 

• Studies to explairi the unstable aspects of 
job selection patterns should be initiated, 
in order to increase the utility of projected 
information concerning labor-market demand 
and to identify means of effecting needed 
changes in selection patterns* 

EVALUATIVE COMMENTS: 

The stated purposes of many vocational education evalua^ 
tion studies indicate that the studies are done to provide 
information for planning purposes. The authors of this re- 
port state that information about job selection patterns 
can Increase the validity of planning that is based on in-- 
formation concerning labors-market demand^ but that data re-- 
quired to compile such information are rarely available. 
The authors acknowledge that further exploration of the job 
selection pattern concept is necessary^ and this document 
makes a convincing case for the desirability of collecting 
the data neGessary for such further testing* 



* * * 
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"Little Woolly Horses and Saber-Toothed Tigers: The, MlsgvMed 
Vocational Education CurricuLum. " iverett Rgginton,^ 
louisvilles University of Louisville/ n.d. , 42 pp. 



DESCRIPTIVE ANNOTATION: 

The intent of this study is "to ejcamine the relation- 
ship between enrollment in vocational education progranis _ 
and attitudes toward self and toward education." Specif i- 
cally, it addresses the question '•Does participation _ in 
vocational education programs result in positive attitudinal 
changes among students?" The author considers this question 
of general interest because there is "a lack of evidence _ 
to support educators' claims about the efficacy of vocation- 
al eduaation programs..." and specifically because "aliena- 
tion among students in vocational education programs. .. is 
at an all-time high even though, positive attitudes toward 
self, toward education and ultimately toward work are un- 
questionablv iinir»ortant goals of virtually all vocational 
education ProfTrams." The specific hypotheses tested are 
twofold: ( 1)' Vocational education students havea more 
nerfative self-image than nonvocational or academic students. 
(2) Vocational education studerits have a more negative 
attitude toward learning than aonvocational or academic 
students , 

The students constituting the sampae were selected from 
public high schools in KentucKy through a proportional- 
stratified random sampling technique. Schools were randomly 
selected from the state's fifteen vocational education ^ 
regions^ and a questionnaire was administered to approxi- 
mately 50 per cent of the juniors and seniors from each 
selected school. Valid responses were obtained from 8145 
students. Two possible biasing factors were mentioned; 



No further citation is given on this unpublished pmprnt 
however, an article based on it appeared in Phi Del ta K appmn 
(April 1978, pp. 533-534) under the title "Is vocationax 
Education Meeting Its Objectives?" 

•"•^The total public high school enrollment in Kentucky 
in 197 6 was 216,009 [U.S. Bureau of the Census. Statistical 
Abstract of the United States^ 1977- (98th editionr 
Washington, D.C., 1977)]. Thus, over 3.7% of the juniors 
and seniors were sampled. ., _ 
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(1) Priric'ipals and guidance counselors, rather than re- 
searchers, drew the samples and administered the question- 
naires i thum, improper aampling may have occurred and/or 
inadequate instruetions regarding the questionnaire may 
have been given to the students. (2) Some of the schools 
randomly eelected ^ere unwilling to participate, thus 
the study sample mmy not have been representative of the 
population . 

AS part of the project directed by Dr. Keith Bayne (pre- 
sumably another University of Louisville researcher) , a 
38-item multiple-choice questionnaire was administered to 
the above--de8cribed sample between September 1 and Decem=- 
ber 15^ 1976. The study here annotated consists of an 
analysis of some of those questionnaire items* The purpose 
and development of the questionnaire are not discussed ; 
the Instrument is appended^ however. 

The independent variable — enrollment in a vocational 
program — was defined by answers to the questioni "Are you^ 
presently enrolled in a vocational program (examples i busi-- 
hess and office, aistributive educatiori/ health occupatione, 
agricultural education, etc.)?" All students who answered 
yes were classifified as vocational students, and the others, 
as nonvocational. {General and college-^preparatory students 
were not separately analysed. 



Four contrQl variables are used in analyses of the hypo-- 
thesesi s©x., age, ethnic origin, and family income. Although 
the questionnaire elicited more precise information, the lat 
ter three variables were artificially dichotomized: 16 and 
under--17 and overr t^hite—nonwhiteF $9, 000 or less per annum-- 
more than $9^000 per annum. 

The data ari© all reported as cross-tabulations, including 
numbers of respondents as well as percentages p missing values 
and missing cases are reported. Tm phi coefficient of 




The dependent variables«-self-image and attitudes toward 
^i^^ ^-^^^ ..r^r-^ hv rftfioonses to specific itesns 



learning some of the time, (i) iiKe learning uii^y 
interest, and (4) dislike learning most of the time 
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correlation Ms used in all analyses and the attained signif- 
icance level is reported. 

Among the results reported and discussed are the follow- 

• ■'•Participation in the vocational f^^^^tion curricula 
is apparently associated with feelings of defensive _ 
ness, although the relationship is slight. [fi = .o^j 

• "The association between VQcatlonal education _ curric- 
ula and feelings of defensiveness holds=-and indeed 
is strong-^-for students 16 and less but virtually 
disappears for students age 17 or more. - .Ob 
and .007 respectively) 

• "Participation in vocational education curricula is 
associated with feelings of inadequacy, although, as 
wis Ihe case with defensiveness, the relationship is 
slight." fJ^ - -02) 

. "When controlling the age, however, the relationship 
^between enrollment In vocational P^^l""'^,^"!^' 
ings of inadequacy] is not sustained for either the 
younger students or the older students. {0 - .02 
in both cases) 

• "There is a significant and Inverse relationship be- 
tween enrollment in vocational curricula and [posi- 
tive] attitude toward learning.' = .05) 

The author concludes "... vocational education programs 
in general are ineffective in changing [the difference in at- 
titudes between voaational education students and their acj-. 
Slf eountelparts]." He states, "Despite the "»"sive invest- 
mSt of funds in vocational education programs and despite 
Sf.acoin?tion that vocational ^^^^^^ion curricula shou^ 
address the problem of negative values and attitudes, tne 
vocatiLal education student still holds hin>self m low es- 
teeiti and tends to treat learning with disdain. 

Egginton then makes several recommendations not linked 
to the lata analyzed in this study. For example, he states, 
"In order to begin to combat the problem of negative atti^ 
tudesffow sell-image, and disdain for ^^^^^'^^^^^^ 
Sonai education students, the larger -°°^^Jy must^prov^de 
the prestige, status, and remuneration which will ^^J^ 

edulatlon «"^^while to the 
he supports views attributea to lerrej, oej.i, 
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and separately administered academic and vocational programs 
serve no useful ands-'i to M, E* Borus, "On-the«^job training, 
apprenticeships I and other non-foTOril mrt^hods of vocational 
training * . . are more etTicionL and O'^'^m! -cH f r^v^M. i vr^ iUnw \ \\0 
vocational training provided by schoolB'*; and to P* J, Kelly ^ 
"The most important job skills which a high school graduate 
can possess are those provided by general education." 

The author of this report is a professor at the Univer'- 
sity of Louisville* It is stated that Kentucky was chosen 
for this study because its "posture toward public vocational 
education programs appears to be representative of the 
willingness of many of the states to establish vocational 
education as a top priority in public education^" and 
appreciation is given to the Kentucky Bureau of Vocational 
Education for permission to analyze the data^ but no 
further indication of sponsorship for this study is given* 

EVALUMIVE COMMENTS i 

Although this study is notable in that^ unlike the 
other studies listed in this bibliography, it addresses an 
immadiate impact of vocational education/ it is fraught with 
so many methodological problems as to make its conclusions 
unwarranted* This is not to say that the conclusions are 
necessarily inaccurate r but rather that they are neither sup^- 
ported nor aontradicted by the study. 

The instrument used in the itudy is appended/ but its 
development is not discussed* No information is given con-» 
cerning its reliability and validity. Even among the 
valid responses collected^ each question analyzed in the 
study was answered by only one to three percent of the 
students. Further^ no rationale is given for judging de* 
fensivenesSf inadequacy/ and attitude toward learning on 
the basis of one questionnaire item each, 

Egginton hopes to answer the question " , , - Does 
participation in vocational education programs result in 
positive attitudinal changes among students?" His design, 
however I does not allow for analysis of change. No argu^ 
ment is made to eupport the aSBUined comparability of 
vocational high school students and all other high school 
students in terms of factors that may affect self --image. 
There is no attempt made to analyse statistically the 
collected data that best suits the question of chanqe^^ 
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namely, the attitudes of sophomore vocational stu|ents as 
compared to those of senior vocational students. 

Another Important problein is that of statistical sig- 
nificance as opposed to meaningful significance. Although 
several of the relationships analyzed were found to be 
statistically significant, the l|rgest coefficient o£ de- 
termination obtained was .0098.=^* This coefficient indicates 
that less than 1% of the variance in any of the variables 
studied can be accounted for by any other variable. The 
meaningfulneas of this small amount of variance is 
questionable. 

Finally, Egginton draws conclusions that contradict his 
stated findings. For example, he concludes that group dif- 
ferences in self-image are sustained for older students 
although his results state that those group differences 
disappear for older students. The result, therefore, 
actually suggests that vocational education may contribute 
to an improved self-image. Further, he makes recommenda- 
tions without clearly stating that they are not directly 
based on his findings. For example, he states that on-the- 
job training is more efficient and cost effectrve than 
vocational education provided by the schools, although 
the study does not address those issues. 



^ ^Although it is stated that juniors and seniors were 
sampled, data tables indicate that sophomores were as well. 
Even if there were too few sophomores for meaningful com- 
parisons, an exploratory comparison could have been made, 
accompanied by a suggestion for further data collection. 

l^see one of the research teKts listed in the technical 
notes section of this bibliography for a discussion of Jhis 
Hill BrieflY, the square of a correlation coefficient is 
Ihf coefficient of detlrmination. When multipled by 100, 
thS ?lluli indicates how much of the variance in one corre- 
JaL Is explained (not necessarily caused) by the other. 
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1975 Career Student FollQv;-- upg Initial Placement s MichaQl 
Quanty, Overland Park ^ Kansas i Office institutional 
Researah, Johnson County Cornmunity College / April 1976^ 
25 pp. (ED 126 999) 



DESCRIPTIVE ANNOTATION i 

Johnson County Conmunity College (JCCC) is required to 
report employment data on its graduates and on students who 
leave v^ith marketable skills to the Kansas State Department 
o£ Education and the Kansas Manpower Utilization System for 
Training, These required data are used to justify expendi-^ 
ture of state and federal vocational funds on career pro-- 
grams* The college takes the opportunity provided by this 
mandated data aollection to acquire additional data for use 
in its on-^going self --evaluation process. 

The 177 graduates and 41 nongraduates with "marketable 
skills*' who left ten aareer programs (Basic Police Training, 
Data Processing^ Dental Hygiene, Drafting, Electrical-- 
Electronics Technology, Fashion Merchandising, Law Enforce^ 
mant, Marketing and Management, Nursing, and Secretarial 
Careers) during the 1974-^75 academic year constituted the 
basic population for this study. Responses were obtained 
from 941 of the population, and also from seven students 
from the program for the hearing impaired, two of whom were 
also enrolled in career programs, 

Most of the data were collected through telephone 
interviews, although the students with hearing impairments 
ware Gontacted personally* Career program coordinators 
talephoned graduates and early leavers from their respective 
programs; when the graduate or leaver was not available, 
the coordinator attempted to obtain information from a 
close relative or work assoaiate* The telephone Interview 
guide is appended. Questions addressed employment status 
(whether employed in program area, in a related area, or 
otherwise employed; whether attending school in a training- 
related area or in another area,^ unemployed; deceased or 
disabled; In the armed forces) ; adequacy of training? and 
gross annual and monthly salary. Additional questions con-- 
cerning the specific programs were asked by some of the 
coordinators (e*g., concerning interest in alumni group 
participation, or in additional coursework) . 

The data are reported as percentages, both for the total 
sample and by program; no statistical analyses were per-- 
formed* In addition, some data collected in the previous 
three fQllow-up studies are provided but no comparisons 
are made* 
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Several results are discussed. Of the fornier students, 
70% were employed in a job for which they had been trained 
and 17% were in a training-related job. For two _ of , thft pro- 
orams, students had to be employed before admission; and, 
excluding these students, 63% and 21% respectively were so 
onployed. Of those employed in training or training-related 
areas, 67% believed their training helped them obtain their 
■iobs Overall, 93% rated the JCCC program as either e^ccel- 
ient'or good, and at least 80% of the students in each pro- 
gram gave excellent or good ratings. The two students with 
hearing impairments who had been in career programs {draft- 
ing, electronics) were employed full-time, but neither m 
the area of training. No interpretations are made or con- 
clusions drawn from the results reported. 

This report was done by a member of the college's Office 
of institutional Research. According to the title page, 
it was prepared for administrative review. 

EVALUATIVE COMMENTS; 

Presumably, because the report was prepared for admin- 
istrative review, it was purposefully left to the adminis- 
i?«tors to interpret results and draw conclusions. More 
fSomltlon would facilitate this effort, for example, why 
sSmfstSeSts do not find training-related employment and 
some stuaencs ao of their programs helped students 

""^^--noiS iyr diploma or the skills they had obtained. 
?f aUow such ISerp^Stations to be made, additional data 
need to be QOllected, 



mnn employers Look Occupational Education. Martin nam 
— - mrger and Har ry E. Woitson. May iyiX; of Educa 

tion of the City of New York, July 1969, 216 pp. 



DESCRIPTIVE ANNOTATIONS 



wflfl the first in a aeries of studies planned to 
provide'! sounflaSe for curriculum redevelopment m the 
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occupational edtiaation prograins of the New York City Board 
Of Education. The objectives of the overall plan to make 
fundamental changes in occupational education were as 
ffollowsi (1) to improve the curriculum; (2) to extend occu- 
pational education to more youth r and (3) to introduce a 
multilevel approach with a variety of time and sequence 
organisations - 

This study was designed to survey employers in an 
effort to determine how to prepare a greater number and 
variety of young people for better jobs. Effective ques=- 
tions to be asked were discussed in workshops-^-^one composed 
of leaders from industry, conmerce^ labor ^ and education; 
the other composed of administrators ^ supervisors ^ and 
teachers in the New York City Schools. 



This project was conducted from February 1 to June 30, 
1968 1 following a long planning period* Prom all the occu-^ 
pational areas in New York City, a sample of curricular 
areas was selected based on diversity, importance, projected 
growth, and adaptability to a variety of school plants aiid 
organisations. The five areas selected— business , health, 
automotive, metal-warking , and electrical/electronics---^ 
include predominantly male, predominantly female, and miKed 
occupations* They include also a range of skill levels 
a balance between white and blue collar, between industrial 
and service, and between stable and rapidly changing fields* 
Firms of various sizes were selected from each category* 
Interviews were held with 105 6 employers, with usable data 
obtained from 994. Because the final sample was not chosen 
through a rigorous sampling plan, but rather was based on 
the criteria stated above and on availability and coopera- 
tion, the authors note that what was done was a case study 
of selected employers rather than a study of a random sample* 
Interviews were held with presidents, managers, personnel 
directors, and production supervisors, 

project staff prepared an open-tended questionnaire and 
interview guide but, after using it, decided that a more 
objective approach would be more fruitful. The instrument 
was then redesigned. The redesigned questionnaire, which 
is provided in an appendiK, basically follows a checklist 
format, although open-tended comments related to each 
question are sought as well. Included are questions about 
job needs, job titles and duties, pay levels, desired 
educational levels, etc. 

The instruments were used during team visits to each 
employment site. Five teams were selected^ each including 
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one teacher and one guidance counselor. Team member iw«re- , 
assigned full-time to the project and were released £»4» 
their teaching and counseling responsibilities for iti , 
duration. An intensive three-week training period wa" 
held, followed by weekly staff meetings during the hmginr- \ 
ning of the data-collection phase, . 

The data analyzed for this study consisted of reseonses 
to the survey questionnaires and impressionistic statements , 
written by the interviewers. The interviewers alfO wrote 
recommendations for curriculum developinent . Quantitative 
data were synthesized into question-by-question suiranaryv tables 
and qualitative data were content-analyzed so they GQ«ld be 
condensed. Excerpts from comments and recomnendations are 
presented, both in the text and in appendixes. 

Findings are presented primarily by occupational groups 
ings in both a detailed chapter and a summary chapter. Among 
the findings discussed are these-. 

• Contacts between schools and employers tend 
to be "hit-or-miss. " 

• Employers indicate that the kind of reading skais 
taught, such as reading work orders and techwal 
manuals, is as important as the amount taught. 

• Although a substantial minority of employers 
consider on-the-job training best, for the ; 
most part employers feel that a school-30b 
partnership is optimal for occupational 

training. 

These and other findings are discussed in detail; many tables 
are Ireeented; and limitations of the data are dlscuised as 
well as their implications. 

This study was funded under the Vocational Edueatien 
Act of 1963 and was approved by the Board o£ Eduoation .of 
the City of New York and the New York State Department pt 
Education. One principal researcher was a Professor of Edu- 
cation at New York University and the other was Asaistftjit ~: 
Superintendent, Office of High Schools, New York City Board .. 
of Education. >, 

EVALUATIVE COMMENTS': 

This study was conducted to collect employer data for 
use in curriculum development. It can thus be viewed in thf * 
context of discovery rather than as an attwipt to verify" any 



.ypotheses about the outcomes of vocational education. The 
•eport is exeniplary in being very thorough i The purpose is 
:learly stated; the relationship between collection and 
malysis of data and the purpose is discussed; the procedures 
md instrumentation are detailed; and limitations of the study 
ire discussed. The authors acknowledge one limitations since 
\o standards exist for determining how much contact there 
should be between schools and employers^ a description of that 
imount of contact is an inadequate basis for appraisal. 



* * * 



Outcomes of Vocational Education in Virginia 1978: A 
Summary of the Follow-up of 1976-1977 Secondary Voca- 
tio nai Education CQmpletions in Virginia . Donald 
Elson. Richmond s Virginia uepartment of Education, 
Vocational and Adult Education, 1978, 18 pp. 



DESCRIPTIVE ANNOTATIONS 

The information reported in this document was collected 
as part of the annual follow-^up begun when the Vocational 
Education Reporting System (VERS) was implemented in 
Virginia during the 1972-73 school year. Data were 
collected from former students concerning employment status, 
on-the--job use of knowledge and skills learned in vocational 
programs, wage rates, and perceptions and attitudes 
toward their vocational programs* No specific research 
questions or hypotheses are listed. 

This study was conducted in January 1978, approximately 
seven months after the close of the school year. The 
population was Virginia public high school students who 
had completed a vocational education program during or 
after the 1976-77 school year or who had left school with 
marketable skills before completing their programs (how 
their marketable skills were determined is nob stated) , All 
business education students and a stratified sample of 
students from other; vocational programs were studied • 
Tables indicating total completions in each area, sample 
size, and number of usable responses are provided* The 
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overall response rate was 55%, representing 10,961 usable 
responsee from the 19,869 students surveyed. Response bias 
was analysed using chi squares on various demographic char- 
acteristics. The data are summarised in a table. There are 
tw statistically significant differences i (1) More females 
than males responded, (2) Proportionately fewer students who 
had left school prior to scheduled graduation returned ques- 
tionnaires than students who were completers* Development 
of the instrument is not discussed, nor is it appended, but 
the several tables and the discussions clearly indicate the 
nature of the questions asked and the form of the response. 
Reliability and validity information is not provided. 

Percentage data for all questions asked are presented in 
"pie charts" and bar graphs for each vocational program 
area. No statistical analyses were done. Comparisons are 
made to the 197 5-7 6 school year. 

Among the findings discussed are the following i 

m 39.15% of the respondents were employed full-time 
in the field for which they were trained, or in a 
related field. 

m 82.7 6% of the respondents were employed full- or 
part-time, were continuing their education, or 
were in the military service, 

• 25% of the respondents were continuing their edu-- 
cation . 

It is concluded that vocational education programs are 
effective in helping former students to obtain employment. 

The Vocational Education Reporting System was developed 
cooperatively by the Virginia Division of Vocational and 
Adult Education and the Division of Vocational and Technical 
Education, Virginia Polytechnic Institute and State Uni- 
versity. The author is a faculty member. Staff of the 
Virginia Department of Education contributed to the data 
analysis and development of the report, 

EVALUMIVE COMMENTS I 

Although a purpose for this study was not stated, it 
can be inferred" from the conclusion that the purpose was to 
assess the effectiveness of Virginia's public secondary 
vocational programs in helping students obtain employment. 
The follow-up system reported here may be capable of 
yielding accurate descriptive information about. what has 
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happened to former students, but it does not collect 
sufflaient data from which to infer causes. In order to 
ascertain whether students obtained employment because of 
the program or some other factors or whether failure to 
obtain employment can be attributed to the vocational 
systam, compariBons must be made and other data collected 
such as unemployment rates in general in the conuTiunityi 
unemployment rates of nonvocational students, and availa= 
bility of jobs. 

Regardless of the quality of the reporting system, the 
response rate for this particular study was too small to 
ensure that the study sample was representative of the 
population. Although response bias was analyzed through 
use of demographic characteristics^ there is no assurance 
that other factors, such as employment status, did not in- 
fluence tendency to respond. 

The ability of students to accurately assess the use- 
fulness of their training is questionable, as is their 
tendency to give an honest and thoughtful response to a 
questionnaire item. Other data could be collected in an 
attempt to verify the conclusion that training is helpful. 
For eKample/ employers" opinions could be sought or 
employed graduates of nonvocational programs could be 
asked whether they found their lack of training to be a 
disadvantage. 

* * * 



"Outcomes of Vocational-Technical--Transf er Programs at 

Community Colleges / Technical Schools, and Similar Types 
of Institutions*" Richard J. Noeth and Gary R. Hanson, 
Paper presented at the Association for Institutional 
Research Annual Forum, Los Angeles, California, May 
1976, 16 pp. (ED 122 892) 



DESCRIPTIVE ANNOTATION 1 

This study, conducted in 1975, was designed to assess 
the outcomes of post^secondary vocational-'technical- 
transfer programs* Specific questions were addressed i 



• What is the present educational-vocational status 
of individuals who began vocational-technical-- 
transfer programs at community colleges, technical 
schools, and similar types of institutions in the 
fall of 1?70? 

• How do students' final prograns relate to their 
occupational eKperiences and to their future occu- 
pational plans? 

• Do individuals working in program-related occupa- 
tions feel that their post-secondary training 
affected the level of their present employment? 

• Are individuals working in program-related occupa-- 
tions satisfied with their present jobs and would 
they go through the same program again? 

In the fall of 1970, 22,342 students from 110 comniunity 
colleges, technical schools, and similar institutions across 
the nation completed the American College Testing Program's 
Career Planning Program as part of the National norming 
"(of abirity, "interests, job choice, job values, etc). 

In 1975 a follow--up sample of 4,350 former students 
was selected from the above groups For this sample, male 
and female students were selected from Business and Market- 
ing, Accounting, Science, Social Science, and Arts and 
Himanities programs; only males were selected from Elec- 
trical Engineering Technology and Auto Mechanics; and only 
females from Nursing programs, A final response rate of 
gQ% — equivalent to 95% of those for whom accurate addresses 
were available-^was achieved with 2594 former students 
from 109 institutions responding. 

The survey questionnaire was pretested eKtensively 
before it was mailed to the student sample. Its development 
is not described nor is the instrument appended. 

The data from all former students who responded to all 
the items pertinent to each analysis were cross-tabulated — 
vocational programs (Business, Auto Mechanics , etc.) by 
criterion measures (present occupation, job satisfaction^ 
etc,). Several tables are provided. 

The following findings related to' each of the four study 
questions are discussed s 

m Among the respondents, 1,489 were employed 
outsiae une home, 234 were continuing their 
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education^ 124 were homemakGr'3 ^ and the 
rest were unemployed for various reasons, 

• Most of those currently emploved held jobs 
for which they were trained (percentages 
are provided by program area) . 

• Over 75% of the currently employed individuals 
agree that they could not have obtained their 
present jobs without their post^-secondary education* 

Among the major conclusions discussed are these i 

• About 75% of currently employed students are em'- 
ployed in their area of training* 

m Students tend to gravitate toward jobs related to 
the educational program they have completed^ 
although their first job may not be directly related, 

• Those employed in training-related jobs are highly 
satisfied with their present occupations and the 
majority would enter their training prograinB again ^ 
if faced with a choice. 

This study was conducted by a research psychologist and 
the Assistant Director of the Developmental Research De- 
partment, Research and Development Division, The American 
College Testing Program. 

EVALUATIVE COMMENTS: 

This study cannot be replicated from the information 
given ^ but since it is a report of a paper presented at a 
conference, that is not to be expected. Cross tabulations 
of data are provided but because statistical analyses are 
not, it is not clear how likely it is that differences are 
due to chance. 

The stated purpose of this study is to assess outcomes 
of post^secondary vocational-technical'-transf er programs* 
It is unfortunate that the reason behind the purpose is 
not stated* It is difficult to further evaluate the study 
without knowing how it was meant to be used. It does give 
an indication of the average (nationwide) success of the 
programs studied , but sufficient information for further 
conclusions is not provided, 

* * * 



117 



A-90 



A Program Review of Secondary Vocation al Education in Ohio-. 
Job Placement and' State Funding . " Ohio Legislative 
Services" Commission, Staff Report No. 126, April 1978 , 
73 pp. 



DESCRIPTIVE ANNOTATIOMi 

This study was done to provide feedback on the results 
of earlier vocational education policy and funding decisions 
and to provide useful information and guidelines for future 
legislative decisions. The study focuses on the job place- 
ment of vocational graduates (at the secondary level) and 
on related issues such as graduate job satisfaction and 
graduate performance as evaluated by employers. The issue 
of how state funds can be better distributed to meet pro- 
gram costs is also addressed but will not be discussed in 
this arnotation. 

Ten vocational education planning districts (VEPDs) were 
randomly selected from the 6 5 that had graduated their first 
senior class by 1976. Four-day field visits were made to 
schools in these districts and one-day visits were made to 
Akron City Schools and to a Joint Vocational School in the 
Columbus area (on the recommendation of the State Vocation- 
al Education Division that they had excellent placement 
programs). One hundred sixty-six employers were inter- 
viewed by phone. Formal guastionnaires were administered to 
vocational instructors and to placement coordinators, as 
well. The sample districts are described in an appendix. 

The questionnaires used in the staff field visits are 
not discussed in the report, although they are appended. 

Findings are presented in narrative form in tables of 
percentages and graphs of trends from fiscal year 1974 througl 
fiscal year 1976. Comparisons on job placement are made 
among VEPDs and among vocational programs. Employers, where 
possible, compared vocational graduates to graduates of 
general programs. 

The results are discussed in great detail, including 
consideration of intervening factors such as availability 
of employment. The following findings are reported: 

• vocational programs similar to those offered by 
technical colleges or proprietary schools do not 
have good placement records. 
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m More than 90% of the graduates indicated that they 
would enroll in vocational education again. 

• Students and teachers eKpressed concern with tech;- 
nical Bkills, whereas employers stressed OTployability 
skills and adaptability, 

• 58% of the employers hire vocational graduates for 
jobs that non-vocational high school graduates 
cannot fill without further training, ^. 

• 76% of employers said they prefer to hire vocational 
graduates rather than general high school graduates. 

• A few students in each sample district said they 
would have dropped out of high school had there not 
been a vocational program. 

Two conclusions reached are that the vocational education 
program in Ohio has been relatively successful in job place- 
ment and that employers, while generally satisfied with 
the skills of vocational graduates^ would favor increased 
communication with school of ficials , The need for a closer 
look at guidance and counseling and at the effects of 
vocational screening is suggested, and other areas for 
further research are indicated. 

In 1977, the Ohio Legislative Services Conmiission 
selected a Subcoinmittee on Legislative Oversight; it sub-- 
sequantly recommended education as one area for joint 
legislative-staff oversight projects. Staff and legislators 
received training and technical assistance in program review 
from Rutgers University* s Eagleton Institute of Politics* 
This study was one of the projects undertaken. It was re- 
viewed and monitored by the Joint Education Coimnittee for 
Program Evaluation. 



EVALUATIVE COMMENTS: 

This study was well defined and thoroughly conducted. 
It is one of the few studies that compare perceptions of 
teachers, students, and employers* It is notable also 
for its discussion of economic conditions in the labor 
market and other variables that affect placement rates. 
Lacking are (1) a discussion of the development of survey m 
struraents and (2) the collection of placement rate data for 
non-vocational graduates- 

* * * 
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A Stud y of Community college Students Who Are Graduates of 
Vocational T echnical and College Prep aratory Curriculums . 
RoHTld J. Horvath. Schnecksviile: Lehigh County Commu - 
nity college, Pennsylvania Department of Education, Bureau 
of Vocational, Technical and Contimilnq Sducation, July 1, 
1973, 96 pp. (ED 096 554) 



DESCRIPTIVE ANNOTATION : 

Although most community colleges maintain an open-door 
admissions policy, there are varying interpretations of the 
concept; some colleges limit admissions to students who 
have completed high school college preparatory curricula, 
whereas others also admit students v;ho have completed 
vocational curricula. The author states that decisions on 
interpretation of the open-door concept are seldom based on 
empirical data. This research focuses on the issue in an 
attempt to provide teachers, counselors, and administrators 
with better insights into the needs and background of 
students (e.g.. Should students take vocational or college 
preparatory curricula if they intend to enter community 
college, and should they enroll in transfer or occupational 
programs?). Specifically, the following three questions 
are addressed; 

• Are community college students who are graduates of 
vocational technical programs similar to or different 
from graduates of college preparatory programs m 
intellective and demographic characteristics? 

• What are the relative completion or non-completion 

records of college preparatory graduates as compared 
with those of vocational technical graduates? 

• What are the intellective and demographic character- 
istics of the vocational technical graduates who en- 
roll in a related college curriculum, in an unrelated 
one, or in a transfer program? 

Six hypotheses relative to these questions are investigated. 

All male students in two Eastern Pennsylvania community 
colleges who could be identified as having completed one 
year or more in a high school vocational technical program 
(excluding commercial, general, industrial arts, and dis- 
tributive programs) constituted the vocational technical 
(VT) sample; 229 were identified. The college preparatory 
(CP) sample consisted of a random sample of 200 male 
graduates prorated among the entry years of 19^7, 1968, 
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196'9r and 1970, using the number of VT entrants as a base. 
This represented 10^15% of the full-time male population. 
ApprOKimately SO percent of the students in the samples were 
graduated from high schools within the sponsoring districts 
of the two colleges (included are four vocational technical 
secondary schools and no comprehensive secondary schools) . 

The primary sources of data were the students' college 
transcripts and personal files ^ which contained high school 
transcripts^ class ranks^ IQ scores^ and American College 
Test (ACT) scores. 

Demographic differences and graduation and program 
associations were tested with ehi-squares. The primary data 
analyses r of group differences^ were accomplished through 
univariate analyses of variance (t-tests and F--tests),^ A 
,05 level of significance was used in interpreting the 
results. 

Although CP graduates scored significantly higher than 
VT graduates on six of the seven educational variables 
tested (the exception was high school rank) and achieved 
significantly higher f irst--semester and final grade point 
averages^ there was no significant difference in college 
graduation rates for the two groups. Therefore , the author 
concludes that this study adds a degree of credibility to 
the community college open-^admissions policy* Other re- 
suits ^ pertaining to the different community college 
programs VT graduates enter^ are discussed and areas for 
further research are suggested. 

The author is affiliated with Lehigh County Community 
College, one of the colleges in the study* Apparently, the 
sponsor of the study was the Pennsylvania Department of 
Education, Bureau of Vocational, Technical and Continuing 
Education, 

EVALUATIVE COMMENTS i 

An important strength of this study is that it had a 
specific purpose and hypotheses related to other research, 
as indicated in the literature review section. Thus, the 
study adds to a body of knowledge rather than being an 
isolated work. Further research is clearly needed, however* 
For instance^ the relative importance and consequences of 
community college grades and community college graduation 
should be investigated. Following that determination, 
correlates of grades and graduation that are more specific 
than the general vocational technical as opposed to college 
preparatory high school curricula should be investigated, 

* * * 
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A ^t-nriv *->^^ Benefits, and Effecti veness of Occu - 

pat ionai E ducatlon T Austin D. Sv?anson. Buffaloi 
Faculty of Educational Studies, State university of 
New York at Buffalo, March 1976, 96 pp. (ED 120 523) 



DESCRIPTIVE ANNOTATION : 

This study was conducted to provide the Board of _ 
cooperative Educational Services (BOCES) for the district 
serving nine component school districts south of Buffalo, 
with "evidence as to what, if anything, was being accom- 
plished [ey its vocational programs] that was beneficial to 
?he region!" Five evaluative criteria had been previously 
established by the Board and its professional staff, and 
two were added for this study i 

• Upon completion of an occupational pragram, 65% of 
the students will indicate that they would take the 
same program again if they were making the choice. 

• 7 5% of the graduates of any licensed occupation 
program will pass the appropriate examination. 

• Within siK months after graduation 60% of the 
graduates available for and having sought employ- 
ment will be employed full-time. 

• Within siK months after graduation, 35% of the 
graduates available for and having sought employ- 
ment in their specialization will be employed 
full-time in their area of specialization. 

. Within six months after graduation, another 15% of 
the graduates available for and having sought employ 
ment in their specialization will be employed full- 
time in a position they consider related to their 
area of specialization. 

« The high school dropout rate will show a decline 
over the years. 

• A high school education will prepare graduates to 
participate successfully in a wide variety of con- 
versation topics and in leisure and civic activities 

This study was designed to address those criteria and to 
accomplish the following objectives as well-. 



m To compare per pupil costs of vocational and non= 
vocational high school programs in the Buffalo^ New 
York, metropolitan area 

• To QOmpare the relative success of vocational 
students and non-^vocational students in terms of 
employment and earnings and also in terms o£ 
selected non^-eoonomie considerations (on the basis 
of the established objectives listed above) 

m To calculate benefit/cost ratios for vocational 
programs 

• To develop decision matrices for evaluating likely 
costs and effectiveness of alternate approaches for 
meeting the district objectives for occupational 
education 

The three phases of the study, conducted during the 
1974-75 academic year, were (1) a program cost analysis, 

(2) collection of effectiveness and control data, and (3) 

(3) cost/benef it^cost/ef f ectiveness analyses. Costs were 
determined on a per pupil basis for the 1972-73 school 
year. Effectiveness data were sought from the 628 partici- 
pants of BOCES occupational programs who were members of 
the classes of 1969', 1971, and 1973 and from a random . sample 
of 422 non-BOCES/ non-college bound students from the 
classes of 1967 (prior to establishment of the BOCES pro- 
gram) , 1969, 1971, and 1973. Usable responses were received 
from 61.5% of the BOCES alumni and 55*2% of the non-^BOCES 
alumni* A table indicating response rates from each class 
is provided* School and background characteristics were 
analyzed for respondents and non--respondents to determine 
whether there v?ere any significant differences between the 
two groups. The only statistically significant difference 
found was between the Occupational Grade Average of BOCES 
respondents and th^t of non-'respondenta ; since the averages 
were 79 and 78, respeatively , it is doubtful that the dif- 
ference hae practical significance* 

The study design and methodology are thoroughly de- 
scribed as are the methods of analysis- For example, the 



Non-^college bound students were those for whom no 
high school credentials were sent to po^t-secondary in-- 
stitutions during or shortly after their senior year, 
ApproKimately 20% of these students did attend a post-' 
secondary institution, but "long after leaving high school." 
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difference between expenditures and costs is defined: 
"Ixpenditures refer to the dollar value of resources 
purchased during a given time period ... [whereas] costs 
refer to the amount of resources consumed during a time 
period." As another eKample, it is stated that average 
teacher salary figures are used to preclude a bias against 
programs that Happened to be staffed with more experienced 
teachers. 

Two instruments were used to collect effectiveness 
data, and both are appended. The School Record Form was 
used to consolidate data taken from school files, including, 
for ejcample, grade average, class rank, IQ, standardized 
achievement test scores, and major course sequence. 
The Alumni Survey Questionnaire has three forms i (1) a _ 
mailed questionnaire for BOCES alumni; (2) a mailed question- 
naire for non-BOCES alumni; and (3) a telephone interview 
schedule used as a follow-up for non-respondents. The first 
two forms of the Survey Questionnaire were identical except 
that the terms "high school" and "Vocational Center" were 
interchanged where appropriate. Three hundred twenty-one 
responses were gleaned from the mailed questionnaires and 
309 from the telephone interviews. The telephone question- 
naire was, in essence, an oral version of the mailed- 
questionnaire I both the substance of the questions and the 
response formats were the same. 

Study findings are reported under several subheadings? 
cost analysis; school record profiles of BOCES and non-BOCES 
students; student performance compared with district ob- 
jectives; cost/benefit analysis; and cost/effectiveness 
analysis— BOCES vs. home school programs; curtailing ex- 
penditures while maintaining effectiveness; and increasing 
effectiveness without increasing total cost. The terms 
graduates, alumni, and students are used interchangeably 
throughout discussion of the findings; for simplicity, the 
30 students who did not graduate are not differentiated 
from those who did. 

Among the findings the following are reported and 
discussed: 

• There are considerable differences between students 
who select BOCES occupational prograins (i.e., those 
who spend some of their school time working xn a 
job related to their training under a cooperative 
arrangement between the school and the employer) 
and students who select specializations m their 
home schools. Although most students m both 
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groups viewed high school as the terminal point in 
their fornial eduaation^ the non-BOCES group inGludes^ 
occupational majors such as business and home economice 
as well as non-occupational majors such as art and 
foreign language . 

m The non-BOCES students scored significantly higher 
(.05 level) on IQ tests and achieved markedly higher 
percentile rankings in reading and mathematics 
achievement tests and in class rank. The differences 
are similar when males and females are analyzed 
separately r although they are more dramatic for 
males ^ 

• When the BOCIS option became available ^ there was a 
decline in the ninth and tenth grade dropout rates 
but an increase in the eleventh grade dropout rate* 
propping out may have been deferred in anticipation 
o£ the BOCES option but, when provided , the option 
may not have been sufficient to keep some students 
in school until graduation ^ 

• For the entire period of the survey ^ 40% of the 
non-^BOCES students indicated that they would not 
re^enroll in the same program if they had to make 
the choice again, whereas 27% of the BOCES students 
would not re-enroll. 

m Of the cosmetology graduates who took the licensing 
eKamination, 94% passed; however, 38 out of the 107 
graduates did not take the exam* 

• 82% of the male BOCES graduates and 81% of the male 
non-BOCES graduates located a full-time job within 
six months of graduation. 

• BOCES and non-BOCES alumni had similar characteristics 
in such matters as conversation topics, leisures-time 
activities, and participation in organizations. 

• In the cost/benefit analysis, the only independent 
BOCES effect on hourly wages was that males earned 
more in their first jobs— an average difference of 

m The Trade Electricity program was the only program 
that had a benefit/cost ratio in excess of 1,0? 
after seven years it was 2*0 and growing* 

This study was conducted by a Professor of Education at 
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the State University of New Vork at BuffaLo for the Board 
of Cooperative Educational Services. 

EVALUATIVE COtmENTS; 

This is a complex study i various aspects of it 
need to be assessed according to different criteria.. The 
report is very thorough and well organized with a detailed 
table of contents and several tables facilitating use of the 
document. 

The study is notable for at least two reasons s (1) The 
characteristics of respondents and non-respondents are com- 
pared^ (2) graduates from two general types of programs are 
compared. It is perhaps uyifortunate that no differentiation 
is made between the non-BOCES students enrolled in occupa- 
tional programs and those not enrolled in this area? ... 
however, the assumption presumably is based on the theory 
that all high school programs are in a sense "oecupational" 
for students who do not intend to pursue further education 
or training. This is an issue that deserves further atten- 
tion, 

A brief discussion of the pitfalls of Cost-Benefit and 
cost-Effectiveness analysis can be found in Attachment C 
Findings of this study should be considered in light of 
those 'pitfalls. 



* * * 



A Study of the Status and Ef f eotivenass of Coopegatlve 
O ffice Educ at ion in New Jeraay^ 1968-69 . Carmela C. 
Kingston. Occupational Research Development Monograph 
NO. 8. Trenton; The Research Coordinating Unit for 
vocational-Technical iducation. Bureau of Oecupational 
Research Development, Division of Vocational Educatiori, 
New Jersey State Department of Education, 1970, 40 pp. 
(ED 060 182) 

DESCRIPTIVE M5N0TATI0N 

This Study was done to detennina the status of public 
secondary cooperative office education (COE) programs in 
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New Jersey and to evaluate the effectiveness of those pro-- 
grams in praparing students for office jobs. It was also 
meant to serve as a model for evaluation of other coopera-- 
tive programs. It was hypothesized that COE students would 
be superior to non-^cooperative office education students 
in the following respects i that they would be employed ■ _ 
sooner, would have more responsible positions, would receive 
higher earnings, would have greater job satisf action, and 
would be rated' more highly by their supervisors. 

The population comprised personnel from New Jersey 
secondary public schools having COE programs during the 
1968-69 school yean principals, business education da-- 
partment chairpersons, cooperative education coordinators, 
COE graduates and non--COE graduates (those who had been en- 
rolled in business classes and who had planned to secure 
office jobs after graduation but . . . had not participated 
in cooperative programs) ; employers of those graduates were 
also included. All the principals, chairpersons,; and 
coordinators and ^'as many as possible" of all the cooperative 
office education students and cooperating business firms 
were contacted. Further information regarding the sample 
is not given, but from the table "Percentage of Replies 
to Each Form" it can be determined that 735 of the 1,112 
employers contacted responded (66*1%) to part of the study, 
although fewer completed the employer rating form and that 
294 out of 400 (73. 5%) of COE graduates and 476 out of 900 
(52.9%) of non-'COE graduates responded to the Job Informa- 
tion Questionnaire* 

Several instruments, with their development based on a 
literature review, were used to collect datai a question-- 
naire concerning factual information, opinions, and view- 
points sent to every New Jersey public high school princi- 
pal; questions comprising background information, partici-- 
pating students, cooperating business firms, the COE 
coordinator, the State Department of Education, and addi- 
tional information sent to each COE coordinator; questions 
sent to graduates concerning their cooperative job exper- 
iences (if they had any) and their present jobs; job-- 
performance rating instruments sent to employers for comple- 
tion. The questionnaires were sent approximately four months 
after graduation. All instruments were in written form, 
and they included rating scales , multiple-choice questions, 
and open-ended questions. None of the instruments are 
appended nor are reliability and validity discussed. 

Data were tabulated using IBM computers; t-tests and 
chi-squares were used to test differences between COE and 
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non-COE graduates. Several tables are provided, mong 
the 34 findings and conclusions reported are the following i 

• Employers were strong supporters of COE. 

• Only 20% of COE students were able to participate 
in school activities. 

• Approximately 8 0% of the COE graduates were employed 
full-time in an office. 

II COE graduates obtained jobs faster than non-COE 
graduates. 

m There was no significant difference between COE and 
non-COE graduates at job entry in weekly gross 
wages received by office workers; however, at least 
one increase in pay was reported by significantly _ 
more COE graduates, and a significant difference in 
weekly gross wages was evidenced five months after 
graduation. 



Both COE and non-COE graduates experienced a high 
degree of job satisfaction. 



• Supervisors indicate that both groups need to 
demonstrate more initiative in job performance. 

• Although COE graduates received higher ratings than 
non-COE graduates in every area of job performance, 
the only ratings that were significantly higher 
(.05 or .01 levels) were . in attitude toward work, 
attitude toward others, and overall ratings. 

Twenty-eight recommendations are made, ranging from use 
of continuous reporting and follow-up systems, to more 
varied job- training experiences, to improved school- 
placement functions. Suggestions for further research are 
made, including a study to determine whether or not more 
secretarial than clerical students are encouraged to 
participate in COE programs and a feasibility study to 
determine the desirability of establishing certification 
standards for coordinators. 

This study was done as partial fulfillment of the re- 
quirements for the author's doctoral degree. Permission to 
do the study was granted by the New Jersey Division of 
Vocational Education. 
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EVALUATIVE COMMENTS: 

The most obvioue problem with this study is the lack 
of smmpling information. The reader is given no indication 
of possible response bias or of why all the responses were 
not used in the analyses (although presumably this was 
the result of incomplete responses) , ^ Further^ although the 
author states that instrument development was based oh a 
literature review/ there is no indication of pilot testing 
or of consideration of reliability and validity. This in- 
formation is necessary to determine how warranted the con- 
clusions are* 

For exploratory purposes and as a model for others ^ this 
study was quite thorough, however. Many factors were investi 
gated from several perspectives, and statistical analyses 
were done. Further research along these lines should include 
consideration of intervening and control variables. 

* * * 



"The Success of vocationally Trained Women in Traditionally 
Male Occupations," David J, Pucel. Paper presented at 
the American Vocational Association Convention ^ New 
Orleans, December 9, 1974, 20 pp* (ED 141 504) 



DESCRIPTIVE ANNOTATION I 

With the passage of laws disallowing sex discrimination 
in employment practices, educational institutions have 
stopped using sex as a basis for determining enrollment in 
particular programs- This study was done to examine a 
portion of the impact of this changing societal concept on 
vocational education. Four basic questions were asked i 

m Are women exercising their right to enroll in the^g 
traditionally male occupational training programs 



A traditionally male occupation was defined as an 
occupation in which at least 80% of the employees are males. 
Such occupations were identified through the 197 0 Minnesota 
Census Data* 
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of the Minnesota Area vocational Technical Insti- 
tutes? 

Do women who enroll in traditionally male programs 
judge the training programs similarly to men en- 
rolled in the same program? 

DO women who graduate from traditionally male 
programs receive or perceive equal benefits in the 
world of work? 

Do employers of the women who graduate from 
traditionally male programs judge them equal to men? 



Only 21 females who graduated in 1972 or 1973 from 
traditionally male occupational training programs and on 
whom the Minnesota Vocational Follow-up system had data 
were identified. The study was continued In spite of this 
small sample because of the importance of the topic. The 
21 females were drawn from eight program ateasi 21 males 
were proportionately sampled from the same areas to com- 
plete the study sample. Data obtained from the Minnesota 
vocational Follow-up System data tapes were analyzed using 
a chi-square technique. Several percentage tables are pre- 
sented as well. 

Among the findings and conclusions discussed are the 
following: 

• The largest female enrollment in traditionally male 
areas was 51% and 46% in chemical technology for 
1971-72 and 1972-73. 

• None of the analyses of male and female judgments of 
their training programs yielded significantly 
different results. 

• The first job salaries of the women were significantly 
lower than those of the men. Pucel notes that data 
collected did not distinguish among the occupations 

in which the respondents were employed. 

• The employment-status differences one year after 
oraduation were substantial, although not statisti- 
cally significant-66. 7% of the males were enflf yef 
in training-related occupations compared to 3B.J.1 or 
the females. 

• The women who received training equal to that of men 
were not as satisfied with the employment they 
entered as were the men. 
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• None of the employer ratings of women were statistic 
cally different from their ratinqs of men* However ^ 
the women were rated lower in terms of quality of 
workf quantity of work, job-related knowledge^ and 
equipment operation. Three possible reasons are 
offeredi a difference in occupations, bias against 
women, and an actual difference in performance * 

This study was conducted by a Professor of Vocational 
and Technical Education at the University of Minnesota 
who is also Director of the Minnesota Vocational Follow-^P 
System, 

EVALUATIVE COMMENTS I 

The author eautions against making weighty decisions on 
the basis of this study because the sample was so small. 
Stilly this has value as an exploratory study because it 
raises questions that can be investigated through further 
research. 

i€ a i( 



Vocational Education Planning District s in Ohloi An ^ 
Ecohomic Evaluation of Foregone Benefits from Limited 
Participation . 1. Ghazalah, Columbusi Division of 
Vocational" Education I Department of Education, State of 
Ohio,. February 1975, 56 pp. 



DESCRIPTIVE ANNOTATION^ 

The author states, ^'The purpose of the economic analysis 
of publicly financed programs is to assist dacision- 
makers in the allocation of a given set of scarce resources 
among competing uses*" This particular study was under-- 
taken because "knowledge about the potential net benefits . . 
to society from increasing the percentage of senior high 
school students who receive job training should assist in 
the appropriate development of vocational education in 
[Ohio]\" The specific objectives of the study were as 
follows I 
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• -To estimate, on the basis of enrollment during the 

fiscal year 1973, Total Net Social Benefits (TNSB) 
from vocational programs at 11th and 12th grade levels 
in each of four sample vocational education planning 
districts 

, TO estimate the TNSB that would accrue from increasing 
enrollment in each sample to 4 0% of the average 
daily membership in those grades 

• Based on sample data, to estimate for each planning 
district the ™SB for the fiscal year 1973 and the 
potential increase from increased enrollment 

• To provide absolute as well as relative economic 
value of the array of vocational education programs 

A sample of four vocational planning districts was 
selected for the study s (1) a single vocational district 
with eleven member schools; (2) a multiple vocational 
district with five member schools; (3) a small joint voca- 
tional school district with six member schools,- and (4) a 
large joint vocational school district with sixteen member 
schools . 

Three sources of data were used in the study r schools 
and school districts,- the state Division of Vocational 
Education,' and published data from the U.S. Department of 
Labor and the Ohio Department of Education. Interviews were 
held with school superintendents and administrators to 
obtain and clarify financial, enrollment, and program data 
about each school and follow-up data on its students. 
Trainees in each school completed questionnaires concerning 
demographic and earnings (from part-time .employment during 
training) variables. The Division of Vocational Education 
supplied unpublished enrollment data. The published data 
sources provided wage rates, labor- force participation 
rates and survival probabilities, and cost- per- pupil- data. 

Although the objectives of vocational education are 
multidimensional, the state-of-the-art in economic analysis 
does not allow specification of a single functional rela- 
tionship that uniquely encompasses all the dimensions. 
Therefore, "Since vocational education is principally aimed 
at increasing the productivity of human resources, this 
study [is limited] to an evaluation of vocational education 
as an investment goal, i.e., in terms of its contribution 
to the objective of raising the output of goods and services 
in the economy as measured by the increase in the earning 
power of participants in vocational education." Costs and 
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benefits to society as a whole are evaluated. In the 
analyseB, "Social Present Value I" is calculated with the 
assumption that without vocational education the student 
would have dropped out of school after the 10th grade to 
enter the labor market. "Social Present Value II" is cal- 
culated with the assumption that without vocational educa- 
tion, the student would have completed the 11th and 12th 
grades in the academic curriculum., separate calculations 
were made for males and females. 

Based on his' calculations, the author concludes that, 
on an absolute basis, all the evaluated vocational 
programs— except child care— have positive social present 
values, making them worthwhile investments. The relative 
value varies with program, with methods of calculation of 
social present value, and with sex. Tables detailing 
these findings are provided. The computed estimates in- 
dicate that increasing vocational education participation 
by 40% would result in a statewide increase in net social 
benefits from $108,918,528 to $326,951,4 34. 

The author of this document is a Professor of Economics 
at Ohio University. The project was supported by funds _ 
from the Ohio Department of Education, Division of Vocational 
Education. 

EVALUATIVE COMMENTS: 

The assumptions underlying the economic anlysis and 
the equations used are discussed in detail. However, the 
data used in the analysis are not. Sources of data are 
listed but there is no indication of how reliable the data 
sources are or how valid the trainee questionnaire was, 
and no indication of how any possible limitations in the 
data would affect the results. 



* * * 
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Vocational TeGhnical and Adult Education ? Student Follow - 

sota County , Flc§rida i Sarasota bounty Board of Public 
Instruetion, May 21, 1976, 68 pp. (ED 124 685) 



DESCRIPTIVE ANNOTATION: 

This is a report of the fourth annual study o£ Voca- 
tional Technical and Adult Education (VTAE) conducted in 
Sarasota County, Florida. It is based on data anlysis pro- 
vided by the local District Data Processing Department 
before the surveys were sent to the state centralized re- 
porting station. Thus, this report was developed to give 
timely data to decision makers before the narrative de- 
scriptions were written. The survey questionnaire addressed 
employment availability, employment status, job classifi- 
cation, quality of training, reason for employment outside 
a training-related area, average weekly pay, source of 
assistance in finding job, and post-secondary education. 
It was hoped that the Information provided would increase 
the ability of decision makers to allocate resources and to 
assist in planning, reviewing, and evaluating the vocation- 
al programs in Sarasota County. 

This survey of 1974-75 completers was conducted in 1976. 
An alert card was mailed to the sample in January and the 
second survey mailing and personal contacts to non- 
respondents were made in Pebruary. 

The survey population was 1025 Vocational Preparatory 
program completers from the Sarasota County Vocational- , 
Technical Center and three high school Home Economies and' 
Cooperative programs. (A listing of specific courses and 
respective response rates is provided.) Many hours were 
spent in deleting from the state computer listings of voca- 
tional students the names of those taking only one course 
and that course for some reason other than job preparation. 
Names of 430 were deleted before the questionnaires were ' 
mailed. The response rate was 63%, which was raised to 671 
by additional responses received after the deadline. Though 
not included in this report, these late responses were ana- 
lyzed with very similar results. 

A pilot instrument for the annual studies had been 
developed in 1971 and changed little until the time of 
this survey. Changes were made to accommodate federal re- 
quirements for collecting veteran and Comprehensive Em- 
ployment and Training Act (CETA) data. The instrument 
appears in the report. The author states that, although 
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validity studies per se have not been done, the high re- 
sponse rate and the lack o£ differences in responses be- 
tween prompt and late responders should increase confidence 
in the data. 

Tabular percentage data for the questions addressed are 
provided for each of the following groups: (1) the total 
survey sample, (2) students enrolled in home economics and 
cooperative programs, (3) total enrollment at the Vocational 
Technical Center (VTC) , (4) VTC students enrolled m health 
occupations, industrial programs, and office occupations, 
(5) secondary students, and (6) post-secondary students. 

. Among .the.-£indings reported are (1) that of those availabl 
for employment in a training-related area, 8 0% were em- 
ployed— 64% of them in their area of training or a related 
field and (2) that, after completing their secondary pro- 
grams, 22% of the respondents went on to post-secondary 
training — about half of them in a related program. 

The annual studies are sponsored by State Department 
officials, who supply survey forms and set the schedule for 
mailing them; this report was prepared by Jim Preston, Pro- 
gram Specialist, Placement and Follow-up, Sarasota County 
Board of Public Instruction. 



EVALUATIVE COMMENTS i 

This report consists, for the most part, of summary 
data tables, i.e., frequency (percentage) charts, with very 
little interpretation of results. A benefit of this format 
is that the report can be pi'oduced quickly, giving decision 
makers access to data before decisions have to be made. 
Another advantage is that specific information sought often 
can be more easily found in a table than in narrative form. 
However, there are disadvantages to this format as well. 
Frequency tables can be misleading in that they present only 
frequencies of occurrences— information that may be crucial 
to the interpretation of differences in frequencies is 
lacking. For this reason, those who consider the "data 
without interpretation" format should consider the disad- 
vantages as well as the advantages. 

* * * 



* * * 
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IV. EPILOGUE: RECAPITULATION AND ASSESSMINT 



This epilogue sunmarizes substantive findings re- 
ported in the descriptive annotations and methodological 
issues discussed in the evaluative comments. It is in- 
tended to give the reader a quick overview of data-based 
claims made about some outcomes of vocational education. 
Readers are cautioned that findings presented here are 
inferences based on a review of the thirty-one studies 
included in this bibliography. 

Several of the studies were designed specifically to 
provide information for describing or for improving particular 
programs, rather than to add to a body of knowledge about 
vocational education outcomes. Still, these studies are 
easily accessible to a wide audience, and their results may 
mistakenly be regarded as widely generalizable by indivi- 
duals who are unfamiliar with the canons of research. It is 
hoped that this enilogue will be helpful in indicating where 
further research is needed and where improvements can be made 
in conducting and reporting evaluation iitudies. Thus, it is 
meant to be helpful both to readers of evaluation reports 
and to evaluators . 

The methodological issues discussed here in reference 
to the annotated studies are discussed more generally and 
in greater detail in Section II above and in the references 
provided at the end of that section. 

It is difficult to draw conclusions based on this set 
of studies because of several definitional problems i 

• The populations studied are often not 
clearly defined. 

• When populations are defined, reasons under- 
lying the choice of definitions are not stated. 

• Reasons underlying the choice of operational 
definitions! of the outcomes under study are 
not stated. 

^An operational definition indicates how an outcome is 
to be measured. For example, intelligence is frequently 
operationally defined by scores on a standardized intelligence 
test. Job satisfaction can be defined by scores on a job 
satisfaction rating scale or by an employee's tardiness or 
use of sick leave. 




After a discusiion of difficulties created by definitional an%l 
other problems, the epilogue turns to methodological issues as 
they apply to sevaral types of outcomes i 

• Employmant and education status 
m Earnings 

• Student satisfaction with training 
m Job satisfaction 

• Job parformance 



populations 

In general^ the populations for these studies are sets 
of individuals viho have been enrolled in vocational programs. 
However, terms are not used consistently in the different 
studies- For eKample, '.'high school vocational graduates" as 
used in one study may indicate those graduates who had taken 
vocational courses in either the eleventh or twelfth grade ^ 
whereas in another study, the term may indicate students 
who completed two credits in a vocational program during a 
school year. 

Not only is there inaonsistency among studies in the use 
of various terms related to the population, but often the 
terms used are not clearly defined. This makes it difficult 
to determine when cQmparability exists* For example , studies 
may include "early leavers with marketable skills" along with 
program graduates without specifying how such early leavers 
are distinguished from dropouts. 

Presumably there are reasons underlying the specific 
definitional choices made. If those reasons were stated, it 
might be possible to determine what populations, although 
different in specifics/ are comparable enough to be considered 
together by those wishing to draw conclusions. The thirty-^one 
studies we reviewed are comparable, with respect to their 
populations, only in that all address those students, gradu-^ 
ates, or leavers characterized as "vocational" by the authors. 



Operati onal Definitions of Outcomes 

In most of the studies the operational definitions of 
the outcomes investigated are implicit, if not explicit. 
For example, the reader can discover how an outcome was de-- 
fined by perusing the questionnaire used in the study* In 
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some cases, however, studies report on an outcome such as em- 
plo^ent In a training-^r elated field without indicating how 
"training-related field" is defined or determined and without 
appending the instrument* Thus, identical or similar labels 
may be used in different studiei for outcomes that are not 
comparable. Table I provides a list of several outcomes with 
definitions preventing comparieons among studies and examples 
of the questions left unanswered when definitions are not ex-- 
plicit. 



Table I 

OUTCOMES OFTEN NOT CLEARLY DEFINED--- 
AND RESULTANT UNANSWERED QUESTIONS 

m Employm ent in occupational program area/in area 
related to occupational program (How are the two 
differentiated?) 



m Job sati sfaction (How is it evidenced^ by self-- 

report or ^aployee turnover? Is it an all-inclusive 
term or subdivided into areas of satisfaction such 
as earnings and working conditions?) 



m. Satisfaction with education or training (What 
is the point of asking? How is it determined? 
Did training help students find or keep a job? 
Would students recomnend program to others?) 



m Self --im age, self -concept ^ self -esteem (As used, 
are these all labels for the same construct? 
How is the feeling measured?) 



m Earnings (Does the term refer to weekly gross 
earnings or average annual income? How can vari 
ous earnings variables be compared?) 

(Continued on the following page) 
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■ Table I (Continued) 

OUTCOMES OFTEN NOT CLEARLY DEPINED — 
hm RESULTANT UNANSWERED QUESTIONS 

Em ployment/UnemployinBnt rates (How are they 
calculated? Is the employment rate based only 
on those available for work?) 



Eflfectiveneas of program (If the study is done to 
determine how etf active the program is or how 
to improve effectiveness/ what is the 
definition of effectiveness?) ' 



Other General issues 

The non-comparability of both populations and outcome 
definitions makes any attempt to draw conclusions from this 
set of studies difficult at the outset. For this reason, some 
other issues related to comparability of studies are relatively 
less important J for example, differences in response rates 
and data collection and analysis procedures. Still, these 
and other methodological issues are relevant to the assess- 
ment of outcomes in any given study and they are not adequately 
addressed in several of these studies. 

One such issue is response bias, which is particularly 
likely to occur in cases where the response rate is small. 
Even the studies that do attempt to analyze response bias fail 
to acknowledge that the factor (s) influencing tendency to re- 
spond may interact with the outcome(s) being assessed. When 
this is the case, bias cannot be analyzed easily. For eKam- 
ple, if the unemployed tend not to respond to employment status 
questionnaires, this bias may not be discovered through anal- 
ysis of demographic variables associated with the respondents 
and those non-respondents who can be reached by phone. 

Another issue is the reliability and validity of the data- 
collection tools or techniques. Few studies discuss the degree 
to which valid and reliable data have been collected. For ex- 
ample, when relatives are asked to report on an "unreachable" 
individual's employment status, there should be indication of 
whether the accuracy of the report was determined. Reliabil- 
ity and validity should be assessed even when the data collect- 
ion appears to be straightforward. 
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A list of guestions relatecl to these and other Issues ;is 
provided in Table II. These questions are Left unanswetfed in , 
most of the reports of studies annotated in this bibliography. 
A discussion of methodological issues related to particular 
outcomes follows the table. 



Table II 



METHODOLOGICAL QUESTIONS UNANSWERED 
111 MANY OUTCOME STUDIES 



• Were questions asked so long after the event (e.g., voca- 
tional training, first job) that one might question ac- 
curacy of recall? 

• Might geographic, demographic^ etc., variables have had 
such an influence that the study results cannot reason- 

" ably be attributed to vocational education at all or can- 
not be applied to vocational education in other settings? ■ 

• Was the study sample Kopresentative of the stated popu- 
lation? Were appropriate sampling procedures used? Wag 
the response rate sufficiently large? Was response bias 
analyzed? 

• Was the methodology powerful enough (sensitive instruments, J 
appropriate data analysis, sufficiently large sample . 
size...) to detect effects of the vocational progrwn? ; 

• Did the research design cover the probable threats to 
validity? What assurance is there that the observed .re- 
sult was a function of vocational education rather than 
of some unmeasured variable (s)? What comparisons vere 
made? . , 

• Were data collected from reliable sources? (How accurate; 
are school records? How accurate is information collected 
from a relative of a graduate?) 

• Was the instrumentation or data collection tool/technique 
carefully developed, pilot-tested, and revised to promote 
reliability and validity? Were questionnaire items 
Mibiguous or leading? 

(Continued on the following page) 
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Table II (Continued) 

MITHODOLOGICAL QUESTIONS UNANSWERED 
IN mm OUTCOME STUDIES 

• Were data collection conditions optimal— questionnaires 
easy to complete, trained interviewers, comparable 
conditions of adrainistr'ation? 

• Are results that are statistically significant also 
meaningful? How much variance in the effect is 
associated with vocational education? 

• Can differences confidently be attributed to vocational 
education rather than to chance? Were data collected in 
a way that permits statistical analysis? Was statistical 
analysis done when appropriate? 

• When data were statistically analyzed, were appropriate 
statistics used? What assumptions were violated and 
with what probable effects? 

• To what degree might factors/tendencies such as social 
desirability, acquiescence, and halo-effect influence 
questionnaire responses? And to what degree might some 
o£ these factors affect tendency to respond? 

• What is the rationale for assigning positive and negative 
values to various points on a scale, since with more re- 
sponse alternatives taken to be positive than negative, 

a random assignment of rankings would result many more 
times in a positive response than negative? 



Employment and Education Status 

Some researchers differentiate "employment in area of 
training" from "employment in training-related areas," and 
others combine the two. Few describe the criteria for 
detennining whether a job is actually in the area of train- 
ing or is related to training, and if so, to what degree, 
some ask the respondents whether their jobs were training- 
related whereas others ask for job titles and use the Dic- 
tio nary of Occupational Titles to make a determination. Some 
report percentages of graduates or leavers with full-time 
jobs only, some report those with part-time jobs as a separate 
category, and others do not indicate their criteria for 
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plaotng an indiviJlual in an employment category. Although 
the studies generally report graduates' employment status soon 
-^f^Qj. ^YiQ students left school (e.g., from six manths to. one yea: 
the data may be collected from six months to eight years 
later (does a job held eight years ago seeni to have the same 
degree of trainincr relatedness to the respondent as it did 
during the firrst six months after gradUrvition?) . 

Some studiea report the percentage of employed graduates 
based on the total number of survey respondents whereas 
others use the number available for employwent as the base. 
Some report emnloyment status by occupational program area, 
some by sex, and others provide no partitions. Few studies 
compare employment status of vocational otudents to any other 
set off students or to any standard at all.'' 

This variation makes it difficult, if not impossible, 
to draw valid conclusions. It dosi appear that on the average, 
a greater percentage of postsecondary vocational graduates 
find anployment and particularly training-related fflnployment 
than do high school vocational graduates. The reasons, how- 
ever, are not addressed? among the possibilities are that 
high school graduates prfef er to seek further training or 



^Reporting rates of employment rather than unemployment 
creates problems of calculation and comparison. People 
who arl Imployed ("worked at all as gai^^ ^"^^1°^! J^" 
the survey week, according to U. S. Bureau of Labor Stftis 
tics [BLS] definitions) are easy enough to count, but in cai- 
culating a rate of employment, one must define and gather 
data foi an appropriate base. This may be all labor force 
participants, only those people ^ho. are "available employ- 
Sent," Sr some other population set. With the base determined, 
however, what comparisons can be made to give meaning to the 
rate calculated for the vocational graduates? Labor force 
statistics published by state and federal govermnent agen- 
cies focus on unemployment (i.e., number of unemployed workers 
livided by the civilian labor force), rather than anployment. 
Unless the base used in computing employment rates for voca- 
tional students is the same as the BLS base used in calculat- 
ing unanployment rates, it is not possible to co^J^f^t the 
rates fot purposes of comparison. Thus, if an ^valuation re- 
port indicates an employment rate of 92% for vocational grad- 
uates whereas the official unemployment rate for^the "levant a< 
qroup is 161, anyone reporting that the unemployment rate of 
vocalional graduates was only half that of th| general popu- 
lation would be in error, and any inferences drawn from that „ 
'conclusion would be misleading. 
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receive more specialised training and ^^'^ therefore more mar^ 
ketable, and that employers consider young adults more desir 
able thin teenagers as employees. It also appears that gradu- 
ates of some occupational program areas are more likely than 
qraduates of others to obtain employment and to obtain train- 
ing-related employment. Further, the occupational program 
areas may interact v;ith level of education. 

some areas for further research are obvious: definitions 
must be specified,- compariions need to be made,, the correlates 
of employment/unemployment status (for example, occupational 
program, sex, educational level) need to be investigated. 

The findings relevant to educational status suffer from 
the same general problems. It appears that more high school 
graduates seek further education or training than postsecondary 
graduates, certainly not an unexpected finding. 

Earnings 

Conclusions about earnings are even more difficult to draw 
than conclusions regarding employment or education status. 
When results are reported strictly as dollar amounts, any com- 
parison between studies, failing to consider the year of the 
study and geographic differences in prevailing wage rates would 
not be meaningful. Not only do many studies report dollar 
amounts without providing other data neeessary for interpreta- 
tion, but they also fail to compare earnings of vocational 
qraduates with those of other graduates or with other stan- 
dards, and they fail to partition graduates by program area. 
Further, some studies report average annual salary, others 
report beginning weekly gross earnings, and others aggregate 
annual earnings. 

Because there is essentially no basis for comparison, no 
general conclusions about earnings can be drawi. In order tor 
earnings to be an appropriate outcome criterion for evaluating 
vocational education, the research community iv.ast agree on nome 
standards of comparison. 

Student Satisfaction with rrainlng 

Students are considered highly satisfied with training 
when they agree with statements like the following: 

• If I had the choice to make again, I would choose 
the same program. 

• I would recommend my program to other students. 
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« My training helped me to get my job. 

• My training has helped me on the job. 

• I was satisfied with my training. 

This outcome was assessed through written or telephone ques- 
tionnaires , sometimes with multiple items, soHetinies only one , 
sometimes using a yes/no format, sometimes using a five-poant 
rating scale. Although some researchers indicate that they 
pilot-tested their instruments, few discuss any aspect of 
instrument development or validation. Without careful devel- 
opment and- testing there is no indication to what degree re- 
spondents' ansiwers are influenced by response sets such as 
Acquiescence, halo effect, or leniency f 3 by ambiguous or un- 
clear statements f or by inadequate response choices (i.e., no 
response fitting what the rBspondent wants ^ • ^n^i^f ^e) . Thus , 
the only conclusion easily drawn is that, when asked, most 
grtdSates will indicate a high degree of satisfaction with 
their training. 

Further research is needed in two areas i i^|*f™J"tation 
and correlates of satisfaction. If student satisfaction with 
tcaining is important, it is certainly important to know what 
fotteri and whit hinders it; it may also be important to know 
what effects it has. To adeguately assess ^-^y/f f °J ^JJf " 
?ng satisfaction, the instruments must be carefully developed 
and their validity and reliability must be analyzed. Much 
^search has gone into the development instruments for 
assessing various attitudes? such research is needed in the 
area of training satisfaction as well. 

Job Satisfaction 

Job satisfaction has been determined in two ways^ through 
use of a published job satisfaction scale and by asking a sang! 
question on a questionnaire or telephone interview, something 
like, "Are you -satisfied with your job?" The issues i-^^assess- 
iSgjob satisfaction, then, are similar to fose in measuring 
twining satisfaction, and the suggestions «f ^^r research 
are analagous. Further, some studies suggest that Pob satis_ 
faction is higher for graduates in training-related ^obs, but 



^Acquiescence is the tendency to make positiYS responses? 
halo eff Set refers to the tendency to say good things abo^t 
eve?y characteristic based, for exanple, on a general positive 
felling about what or who is being rated, or on one 
pllntf and leniency is the tendency to give the more positive. 
rating when unsure of which rating to give. 
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others do riDt distinguish those with training-related jobs 
from those w^ith other jobs. This, theii, is an area needing 
further reaearah. 

There is a body^ of literature on job satisfaction,^ indi- 
cating that it is a multidimensional concept anienable to aspess 
mant from various perspectives. For example, job satisfaction 
may be inferred, in part, from turnover rates and use of sick 
leave. Vocational education researchers would do well to ex- 
amine the literature and to do further research on what kinds 
of measurement are most appropriate for evaluating vocational 
education against a job-'Satisf action criterion. 5 



Job Performance (Attainment of Occupational Conipetencies ) 

One wa/ to evaluate job performance is to ask employers 
hoxtf their employees perform on the job^ the only technique 
(beyond asking the graduates whether their training contributed 
to their job performance) used in any of the thirty«-one studies 
presented in this bibliography. This is not to say that, per-' 
formance testing is not donef but it does not appear to be 
done often in vocational education outcomes evaluation. 

Only a few of the thirty--one studies reviewed in this ^ 
bibliography sought employer perceptions/ and those that did 
are fraught ^ith problems limiting the degree to T^hich their 
results contribute to knowledge about the job performance of 
vocational education graduates. One problem is the use of 
presumably untested, unvalidated rating scales. As discussed 
in the section on training satisfaction, without careful de- 
.'elopment and testing there is no indication to what degree 
responderita' answers are influenced by factori unrelated to 
the substantive content of the rating scale*. 



See^ for instance, Robert P. Quinn and Martha S. Baldi 
de Mandilovitch. Educatio n and Job Satisfaction: A Question - 
able Payoff , Washington", DTC U. S. Department of Health, 
Edrycatlon', and Welfare, National Institute of Education, March 
1977. This document is annotated in the Appendix of Key Re-- 
lated literature, 

^See Kenney E. Gray, Marie Abram, and Floyd jMcKinney, 
V Qcafcional Education Measures; Instruments to Survey Former 
gyaerits a^a Their Employiys- Golumbusi The National Center 
£c>F^R?seatch in Vocational Education^ The Ohio State Univer- 
sity, 1978 1 for a review of some of the instrurnents available. 
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Another problem is that data indicating the extent of 
employers' experience with vocational graduates have not been , 
colleQted (for example, total number hired, percentage of 
einployees from vocational programs, how long vocationally 
trained employees have been used) . Further, the data that 
have been eollected have not been verified. For example, 
hiring-preference data consisting of questionnaire responses 
have not been verified by inspection of personnel records. 

Finally, comparative data hove not been collected. Some 
giiestionnaire items asked whether graduates of vocational pro- 
grams were better employees than graduates of general programs, 
but none compared performance ratings of vocational anfl non- 
vocational graduates . 

With all tHese difficulties, the only conclusion that 
can be drawn from these studies is that employers tend not ^ to 
express flis satisfaction with employees who are graduates of 
^tional programs. Further research is needad^to overcome 
the ptoblems aiscussed above. More studies should address 
multiple factors of performance to discover whether vocational 
education affects some factors but not others. It cannot be 
overemphasized that there are several problems inherent in the 
use of rating scales that can be summarized by the statement 
that a rating scale often reveals more about the rater than^ 
about the substantive content of the scale. Thus, it is. very ■ 
?mpSJtant thafrating scales be carefully derived and thoroughly 
tested and that verification be done where possible through 
use of other assessment techniques. 



Other put cpmes 

Other outcomes addressed in some of these studies include 
leadership, voting behavior (as a proxy for societal partici- 
oation) , self-image, academic standing, success of women in 
traditionally male jobs, and dropout rates. These outcomes 
were not addressed by enough studies to permit even an attempt 
at synthesis of findings. In most cases, the research suffered 



from the limitations discussed above in reference to particu- 
lar outcomes 



^See Attachment "C," Table II, which indicates types of 
data that can be collected for evaluation purposes. 
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I . INTRODUCTION 



This Appetidiic presents bibliographic citations and anno- 
tatiofts of literature contributing to a better understanding 
of voeational education outcomes. It supplements the mam 
body o£ the bibliography, which consists of empirical out- 
come studies, by providing background information on voca- 
tiotial educatlori/ evaluation methodology, and other topice 
related to vocational education outcomes evaluation. 

Bntriea are listed alphabetically by title within the 
following eategories: 

Category 1: Data-Base Reports 

category 2: Review and Synthesis Papers 

category 3s Evaluation Methodology 

category 4: Empirical Studies Focusing on Voca- 
tional Education Topics Other Than 
Outcomes 

category St Miscellaneous 

Data-b?sp re ports included here provide statistics and 
otherTnfHHltion pertinent to national studies of vocational 
education outcomes, including explanations of how the data 
were ccllected. These reports are utilized by researchers, 
evaluators, practitioners, policy makers,, and the general 
public. Illustrative of Category 1 are Learning a Living 
Across the Nation, the "Project Baseline" series produced 
tor''^hr^atIHHar'lnstitute of Education (NIE) , and Mational 
r.r,nqit.ndinal Stud y of the High S chool Class of 1972, surveys 
Spniora^W^e National Center for Education Statistics . 

and svnthesis papers provide an overview of vari- 
oas i§iS-57%Sationil M Satiln evaluation. They help 
Identify critical issues, report diverge ^^^^^'J^^' 
elSions Of evaluators, and list bibliographies of *ejevant 
?ttJrature An example of the type of document included m 
JilSgiry f • i . Prgcrram Evaluati on in Vocational Educat i on^ h 
rSJiIS a paper-iloducir for the Educational KeSBurce Inffor- 
iill'cenSr (EllC) Clearinghouse on Career Education as 
part of its information analysis series. 



Deeuwants that discuss eva Itaa t i o n me thodo logy i modals, 
frainewaC'kSr and ays terns appliQahla to vocational aduaatlon 
mra Included in Category 3» Because there is a body 
of litaratura In this araa^ only a aample of publications 
fraquantly oltsd or highly reaQsroended by vocatlpnal eduaa-* 
tlon admliitstf ators and researchers was seleoted* An e^Miple 
is Eyaluatlng Vocational EduGationi Polloieg and Plans for 
tha a v^agiaty of niethoda for measuring 

aduoational output and addresses analytical problems involved 
ir^ tha avaluatton of vocational education 

As aKplainod above, the main body of the blbliograp!' f 
provides desoriptiv^^ and evaliiativa annotations of empirical 
vtfaatlonal aducation outcome studies i reports of empiri lal 
invent igations concarning the consaguenaes of vocattonal 
programs* Entries in Category 4 of this appendisc, in cou-^ 
t^aat, are restricted to evalaatton studies of other aspe cts 
ef voeat ional education # namely contexts student charact ir- 
rs t;J 00 resSurees { inc lud ing facilities) / goals ^ and pro ^ 
easaaa* One example is What is the Role off Federal Assis - 
tanoa for Vocational Education ? 

Pinally, Category 5 Includes miscellanaous docuinents 
which discuss vocational education or th^as vocationally 
related from a variety of perspectivaB t For eKample^ youth 
CTployntent is studied not only as an outconie of vocational 
programs, but also in terms of government-subsidised jobs, 
includad in this section is a study sponsored by the NIE, 
Educatio n and Job Satisfaotioni h Questionabla Payoff , 
Evaluation reports on special needs subpopulations # bbk 
fairness I career education , economic aducationr and other 
genaral/acadwiic education programs are also repreaanted in 
this sactlpn. 



As im the case with other publications listed in the 
appandlK/ this book also contains material included in other 
categories* No multiple listings are made^ however • 
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A.S indicated previously # citations and annotations are, 
presented alphabetically by title within the respective cat- 
egories (with initial articles "A," "An," and '^The" ignored j 
in plaoing the publications in alphabetical orfler) , In con- 
trast to the annotations presented in the main body , of the 
bibliography, the appendix annotations are descriptive only. 
In most cases, the document's purpose, major themes, conclu- 
sions, and author identities are reported. 

Humbers designated "ED," where provided in the citations, T' 
re£er to the accession number sequentially assigned to docu-^ | 
ments as they are processed into the ERIC system and pub- _ ;| 
lished in the monthly abstract journal Resources in Education r 
of Nil, These documents, except as noted in ResourceB in ;. 
Education , are available in either microfiche or hard copy 
(paper) from! . \ . ' '-'M 

ERIC Document Reproduction Service (IDRS) 
p. 0. BOX 190 
Arlington, Vk 22210 

Many university libraries have collections of ERIC microfiche. 
When no ED number is given for a particular document, it pre- : 
sumably has not bean entered in the ERIC system. 

Following the annotations are three consolidated indexes 

which provide a listing of all categories, arranged alphabet- 
.ically by: 

• Author (cross-indexed to title and category) 



m Project Identifier /Publisher (cross-indexed to title ; 

and category) M 

• Title (cross-indexed to category) ;| 
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APPENDIX OP KEY RELATED LITORATURE 



Category 1: DATA-BASE RJlT^ORrs 



LIST OF TITLES Page 

Career Thrasholdsi A Longitudinal Study of the Eaucatlonal 166 
~ and Labor Market Experience of Male^ Youth/ Voliiine I\ 

Herbert S. Parnee and others. Center for Human Resource 
i Research, The Ohio State University^ 1970. 

Career Thresholds % A Longitudinal Study of tha Educa - 16E 
tlonaT^nd Labor Market EKparience of Male Youth . " 
Volume 2l Frederick A. Zeller and othars. Center for 
Human Resource Research, The Ohio State University, 
1970, 

C areer Thresholds g A Longitudinal Study of the Educa - 168 
tional and Labo r Mark et EKperiehce of Hale Youth . 
Volume 3l Andrei^ l'. Kohen and Herbert S. Parnes, Center 
for Human Reaouroe Research, The Ohio State University, 
May 1970. 

Career Thresholds g A Longitudi nal Study of the Educa - 169 
tional and Labor Market Experience of Male Youth . 
Volume 4 . Andrew" I, Kohen. Center for Human Resource 
Research, The Ohio State University, January 1973* 

Career Thresholds; A Longitudinal Study of the Educa - 170 
tional and Labo r Mar ket EKperience of Male Youth . 
Volume S .^ Paul Andrlsani and Andrew I* Kohen. center 
for Human Resource Research, The Ohio State University, 
April 197S. 

Career Thresholds! A Longitudinal Study of the Educa - 170 
tional and Labor Market EKperience of Male Youth . 
Volume^ . Andrew I. Kohen and others. Center for Human 
Resource Research, The Ohio State University, March 1977. 

Dual Careers 1 A Longitudinal Study of Labor Market Experience 171 
of"' Women. Volume 1 . John R. Shea and others. Center 
for Hmnan Resource Research, The Ohio State University, 
May 1970. 
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^earning a Living Aaross the Nation . Volume 1. Pro ject 1 7 3 

Baseline SsGond (Sic) Natloiial Report, Bas^ffi^a Ye ari 
T^T-lS'^l (Fiscal Year 19 71). Arthur M. Lee, iqorthern 
Arizona University^ Novarnber 1972. 



Learnirig a Living Across the Nation. Volume 2. Proiaet 173 
lassLine Seaond National Report , Baseline Year i 1971 * 
1&72 (Fiscal Year 1972) / Arthur M. Lee and Robert 
Sartln. llorthern Arizona University^ November 1973. 

Learning a Living Across the N ation, Vo lume 3. Project 174 
Baseline Third Na tional Report. Baselin e Year 1972- 
1973 (Fiscal Year 1973) ^ Part 1 and Part 2/ Arthur 
Lee- Northern Arizona University, Novembe? 1974, 

Learning a Living Across the Nation. Volume 4. Project 17S 
Baseline Fourth National Repqr^^ Baseline Year 1973-^ 
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APPEITDIK OP KEY RELATED LITERATURE 

Catecrory Is DATA-BASE REPORTS 
Citations and Ann otations 

Career Th aresholcla i A Longitudinal Study of the Educational 
^ and Lahor narket Experlenc es''D £ Male Youth. VoluTne l. 
Herbert S. Parnes, kobaft~C. Hiljus, and Ruth S. Spitz. 
Columb-us: Center for Kuman Resource Research, The Ohio 
State University, 1970, 276 pp. 

Data reported and analyzed in the lix volunies entitled 
Ca reer Thresholds were collected from Interviews with a 
national sample selected to be representative of approxi- 
mately 16 million young men in the tJ. S. civilian 
institutionalized population who were in 1966 between the 
ages of 14 and 2^. The general purpose of these studiei 
is to contribut.Q to an understanding of the factors that 
influence labor' market "success" for young men of this 
age group . 

In addition to educational and labor market experiences of 
the young- men, the study eKamines their command of occupa- 
tional information, their job attitudes, and their edu^^ational 
and occupational aspirations. Among the explanatory var- 
iables investif^ated are age, family structure, financial 
condition, s]-i.Us, health and physical conditions, school 
experiences, anf. certain environmental variables such ae 
size of labor force and unertiployment rate. ,^ 

The data reported as a result of the first round of inter- 
views in 1966 were subjected to tabular analyses as an 
initial step in identifying the variables most . influential 
in explaininff variations in aspirations and labor market 
experiences. ' Chapter 2 of Volume 1 describes the demo- 
graphic and social characteristics of the group, examines 
the distribution of the characteristics, and considers some 
of the interrelations among them. For instance, blacks were 
found to fall taelow whites on all of the socioeconomic 
status indicators—they were less likely to be enrolled m 
the colleffe pretiaratory curriculum, they had completed 
fewer years of school, and they were less likely to have 
had vocational training outside of the educational system. 
Youth with rural backgrounds were less likely to be m 
school than those from urban areas. 



^a^tora are invssfcigated in Chapter 3 that appear to differ- 
^^mi^^Mtm young meri who are employed from those who are not. 
i , lower iinamploypient rate of young men In their 

kw^nties than that of those in their teens was related to 
tha likelihood that the older group v/as better educated, 
married, white, and non-atudents. 

Chapter 4 examines the types of jobs the young men hold, 
hours worked per ^\reek, and rate of pay. For the young ir.en 
20-24 years of age not enrolled in school, hourly rate of 
pay was found to be nearly uniform among the m^ajor occupa-^ 
tion groups and was found to be positively related to the 
number of years of school completed, to the extent of occu^- 
pational training outside regular school^ to good health, 
and to the si^e of the residence community. 

Chapter 5 describes and analyses findings from a section of 
the s,urvey concerned with measuring the respondents' "know^ 
ledge of the world of work* Scores from a sainple occupa--^ 
ticn information test (reproduced in App#ndiK T of Volume 1) 
ar€4 related positively to extent of eduoation# measured 
intelligence, and socioeconomic status* When educational 
attainment is controlled, those with high scores on the 
occupational information test receive higher wages than 
those with low scores* 

Chapter 6 explores attitudes of the employed youths. toward 
their jobs. The majority — 90 per cent of the i^hites and 85 
per cent of the blacks— expressed positi^re feelings about 
their jobs. Edt^rational and occupational aspirations re- 
ported in chapt^y^ i ^;n.dicate that young men between 14 and 
17 still enrol^sA*^;^ aohool have very high educational and 
occupational gtMls* Sixty per cent say they want to obtain 
four or more yoars of college, and 70 per cent want at least 
two years of college* Almost half say they want to be in 
professional or technical occupations by age thirty* 



* * * 
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This volume reports finaings of the second round of National 
Lonqitudinal Survey interviews of young men and is charac 
terlzed as a proqresa report designed to describe the 
^aSllude an/patterns of change that have occurred between 
th4 first and second interviews— changes m school enroll- 
ment status , labor force Participation, unemployment 
experience, occupational mobility, and educational aipira- 
tions. Summarized findings include the following i 

• Downward revision of aspirations was most 
prevalent anoner those whose 1966 educational 
goals V7ere highest and whose, socioeconomic 
indicators appeared least supportive. 

• While youths suffer high rates of loblessness 
and the problem is especially severe for blacks, 
a substantial portion of the unemployment re- 
ported is for those still in school before 
their transition to full-time vfork. 

m Hioh rates of occupational movement may reflect 
inaopropriate early job choices but may also 
refiect^a "healthy flexibility" that permits 
adjustments leading to improvenent in status. 



rareer Thi-esholds; A Longitudinal ptudy of the Educational .| 
"^•"lEa Labor Itorket Experiencg^ojjli^Youth. Volume 2. .| 
?5ecSerick A. Zeller, John R. Shea, T^ndrew I. -Kohen, and ;-^| 



Jack A. Meyer. COlumbusi Center for Human Resource 
Research, The Ohio State Uiiiversity, October 1970, 16i 
pp. (ED 047 104) j 



* * * 
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career Thrr-th-i ^° • a T.nn^ltnclinal Study of the Educational 

^—TTKEhiH^a Herbert sTpiHis. Columbus r ^^nter 
fSr luman Resource Research, The Ohio State University, 5 
May 1970, 165 pp. (ED 054 336) 

Findings of the third round of interviews conducted in 1968 
arf sSLari.ed in Volume 3, and ^^^^^^^^^^ 
labor market status occurring during the two years between 
the li?6t (1966) and third (1968) surveys Jescribed^^^^ 
The report states that since the external en^^^onment ^the 
v'L; L;v«t=--.ms fairly similar at the two pomte m time, 



''V 



it is ^3^\}i^ed that ohanges in status of the members of the 
sample ara primarily attibutable to the changes in their 
personal charaateristias . For instance, the effect of 
"aqing"-^two additional years of schooling and/or work and 
life exi^erience — may have profound effects on their depth 
of understanding and seriousness of purpose.. 



Car eer Thresholds i h Longitudinal Study of the Educa tional 
a nd Labor Market^Kperience of Male Youth. Voliime 4 , 
Andrew I. Kohen, Columbus i Canter for Human Resource 
Researchi The Ohio State University/ January 1973 ^ 208 
pp. (ED 074 280) 



During the three^-yeair period between the 1966 and 1969 
interviews^ a third of the young men changed their school 
enrollment status; a fifth changed their raarital status ^ 
arid a tenth of the students and more than a fourth of the 
non-students changed county (or Standard Metropolitan 
Statistical Area) of residence at least once* 

While it was noted that high school graduates were more 
likely than dropouts to have favorable labor market experi-^ 
ences, it was acknov/Ledged that both dropping out and 
unfavorable labor tnarket experiences may be attributed to 
social and personal disadvantages. 

Following are some of the differences found between graduates 
and dropouts I 



• Graduates had lower rates of unemployment 

and were more likely to be employed full time* 



In examining changes in the educational and occupational 
goals of the young men Gontinuoualy enrolled in school lince 
the original interview, nearly one^half had changed their 
educational goals and three^fourtha had revised their occu- 
pational aspirations* Changes in occupational preferences 
were found to occur mainly within i not across, occupational 
categories , 

* * * 



Career Thresholds i A Longitudinal Study of the Educational 
and Labor HarketEkperlence o f Male Youth, Volume 5 . 
Paul Andrisani and Andrew I. Kohen, Columbus i Center 
for Human Resource Research, The Ohio State University, 
April 1975, 92 pp, (ED 111 961) 

This report explores the impact of collective bargaining 
coverage on the 1969-1970 labor market eKperiences of a sub^ 
set of member 5 of the gr -up who were 17 to 27 years of age, 
not enrolled in school, .nd employed slh blue-collar workers 
in 1969, 

The study led to the conclusion that collective bargaining 
coverage had a iubstantial net impact on the hourly earnings 
of blue'-collar young men both white and black. Among blacks, 
those who are unionised experience less unemployment than 
those who are non--unioniEed. 

* * * 



Career Thresholds i A Longitudinal Study of the Educational 
" and Labor Market EKperiences of Young Men> Volume 6 . 
Andrew I. Kohen, John" T- Graeso, Steven C, Myers, and 



Chapter 2 reports findings on goal formation and revision 
behavior^ concluding that goal development by young men is 
an adaptive process developing over tlme- 

Chapter 3, '*The Labor Market Effects of Investr nt in Human 
Capital," further explores the connection betwe education 
and work* The findings suggest that schooling, formal train- 
ing, and on-the-job training are all important for achieve- 
ing labor market success during the early career period. 
However, measures of high school or college quality were not 
found to be related to labor market achiavements ^ nor did 
the data indicate the existence of a credentials effect or 
the advantage of one high school curriculum over another. 
The study found (p, 192 — ^Erratum) ''no evidence . . - that 
graduates of high school vocational curricula fare better 
than other graduates in terms of starting rates of p^y." 
The group 'b occupational mobility is explored in Chapter 4; 
youth unemployment and the various kinds o£ job separations 
are analyzed in Chapter 5; and determinants and consequencai* 
of serving in the armed forces during the Vietnam war are 
considered in Chapter 6* 



* * * 

* * * 



Dual Careers: A Longit; gndl Study of Labor Market 

E>£periehce o^ "Women/ Volume 1 . John R, Shea, Ruth S* 
Spitz7 and "Frederick" A, Zeller. Columbusi Center for 
Human Resource Research, The Ohio State University, 
May 1970, 288 pp. (ED 043 755) 



A--144 



Of particular interest in the study of th: . .jroup are axpa^ 
riances and prohlems associated with re-en ary into the labor 
force by married women after their children no longer require 
their continuous presence at home. 

Among the survey findings reported are the following s 



a Women who had never been married experienced 
net upv/ard mobility from their first to their 
current job. 



• Women who were^ or had beenr married start 
lower on the occupational ladder and more of 
them experience downward mobility than exper^ 
ience upward mobility* 

• Over 2 0 per cent of the white women and 33 
per cent of the black women vrere married be- 
fore age 18. 

m Of those married before 18, a third of the 
whites and ju.it over half of the blacks com- 
pleted less than four years of high school. 



Other volumes entitled Dual Careers and reporting data from 
subsequent interviews in the original five-year study in^ 
elude the following^ 

Dual Careers: A Longitudinal Study of La bor Marke t 
Experience of ^^om^n , Volume 2 « Pookon Kim and others, 
Columbus 1 Center ^f or Human Resource Research^ The 
Ohio State University^ September 1972^ 131 pp. 
(ED 068 713) 

Dual Caree rs; A Longitudinal Study of Labor Market 
Experience ~o f Women . Volume 3 , Carol L. Jusenius and 
Richard L. Shortlidge , Jr . Columbus : Center for Human 
Resource nesearch, The Ohio State University ^ February 
1975, 1Q7 pp, (ED 108 053) 
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Learning A Living AcrosB the N ation. Vo lume 1. Project 
^ Basalina gec ond [e ic3 National Report. Baseline Yea r^ 
1970--1971 (FlscaT^ar 1971) , Arthur M. Lae, Flagstaff i 
Northarn Arizona University / November 1972, 454 pp, 
(ED 695 309) 



This first report in a series of five published under the 
title Learnlncf A Living Across the Nation uses Project 
Baseline data to describe and analyze the status of voca-- 
tional education and manpower training programs between 1970 
and 1975. This series was prepared for the National Advisory 
Council on Vocational Education as a result of a request for 
information by the Appropriations Committea of Congress - 
The statistical data reported are drawi primarily from of^ 
ficial state records combined with statistical data from 
the U, S. Census Bureau and the U. S. Department of Labor 
and include data from state education agencies not required 
by the U* S. Office of Education. Volume 1 reports data 
for the 197 0-1971 school year* 

^ * ^ 

Learning A Living Across the Nation. Volume 2. Project 
Baselin e nacond National Report. Baseline Yean 1971 - 
197 2 (f iscal Year 1972) : " Arthur !% Lae and Robert 
Sartfn. Flagstaff ^ Northern Ariiona Univarsity^ 
November 1973, 499 pp- (ED 095 310) 



Volume 2 is characterized as a companion volume to the first 
in the series of Project Basaline reports, 3ince the first 
and second volumes were issued simultaneously^ many of the 
tables in the second volume contain firsts-year data not 
found in Volume 1» Vocational education enrollment ^ costt 
and placement data for the 1971-1972 school year are re- 
ported according to secondary/ post'secondary ^ and adult 
ievels of instruction by program areas. 
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L earning a Living Acrosa the Nation. Volume 3. Project 
' — B aseline Third N ational Report. Baseline Year; 1972- 
1 973 (Fiscal Year 1973 ) .' Part 1 : N arrative Report . _ 
Arthur M. Lee. Flagstaff: Northern Arizona University, 
Novamber 1974, 108 pp. (ED 099 684) 

L earning A Living Across the Natio n. Volume 3. Project 
Baseline Third National Report. Baseline Years 1972- 
1973 (F iscal Year 1973), Part 2; Statletical Almana c. 
Arthur M. Lee. Plagstarf: Northern Arizona University, 
November 1974, 210 pp. (ED 099 685) 



Volume 3 of the Project Baseline reports covers the school 
year 1972-197 3 and is divided into a narrative report of 
findings (Part 1) and a statistical data report (Part 2) on 
which the narrative is based. The first chapter of the 
narrative report contains a aummary of significant findings 
based on the data, suggested implications for policy making, 
and these recommendations : 

• New federal legislation for 197 5 should 
include provisions for states to r? .{uallze 
vocational education opportunitie'. for all. 

• Federal spending priorities, as they affect 
vocational education, manpower training, un- 
employment, and we'lfare, should be r»"e,i;,amined . 

/ 

• Congress should Include In the legi.slAtAOii_ 
for 1975 a set of vocational education defi- 
nitions, allowing for fleKibillty and, at the 
same time, providing for national standardiza- 
tion of terminology. 

• A new reporting system should be established 
for both vocational education and manpower 
training in which data elements in machine 
readable form can be transmitted from local to 
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Learning A Living Across the Na tion, Volume 4> Project 
' Basel ine Fourth National Report. Basela^ne Year: 1973 - 

1974 (Fiscal Year 1974). Part li Narrative Report . 

Arthur M. Lee. Flagstaff: Northern Arizona University , 

DeGember 1975, 144 pp. (ED 120 470) 

Learn ing A Living Across the Nation. V olume 4> Projeot 

Baseli ne Fourth National Report. Bas eline Year: 1973- 

T57T^(Flsaal Year 1974). Part 2i Statistical Almanac , 
Arthur M. Lee and Dorris Fitzgerald, Flagstaff: Northern 
Arizona Unierstiy, October 1975, 111 pp. (ED 120 471) 



The fourth year Project Baseline data published in two parts 
provides a narrative report in Part 1 and a statistical data 
report in Part 2. A question and answer format is used to 
discuss vocational education enrollment ^ expenditures, 
placements, instructional personnel, support services^ and 
manpower data for the 1973--1974 reporting period. Chapter 3 
reviews the impact of vocational education research and 
demonstration efforts, including a review of 28 state- 
administered research projects (Part C) , 16 federally 
administered exemplary projects (Part D) , and 12 state- 
administered exemplary projects, all of which have had 
extended use. 

While considerable improvements in the reporting of voca-^ 
tional education rl^ta over the previous years is noted, per-- 
sisting problems resulting from the lack of a nationally 
standardized set of definitions and inconsistent handling 
of data are pointed out, " review of information systems 
currently in use in Gach v .te is included along v/ith a 
report of a special task iorce effort to bring uniformity 
to definitions of key terms used in reporting data', includ- 
ing the term "vocational education." 



* * * 



Learning A Living Acroes the Nation. Vol\:^a 5, Project 
"Baselina Fifth National Report. SaBeline_Year s 1974 -' 
1975 (Fiscal Yea^ 1975")^ Part li Narrative Report , 
Arthur H." Lee. FlagBtaff i Northern Arisona University / 
Noveinber 1976, 188 pp. (ED 137 505) 

Learning A Living Across the Nation. Volume 5, Project 
Baseline Fifth National Report . Saselrhe Year 1974 - 
1975 (FTscal Year^ 1975) . Part 2% Statistical Almanac . 
Arthur M. Lee and Dorris Fitzgerald. Flagstaff: 
Northern Arizona University, November 1976, 136 pp, 
(ED 137 551) 



The f if th^year Pro ject Baseline report is divided into two- 
parts, with the narrative report contained in Part 1 and the 
statistical data contained in Part 2* A question and answer 
format is used in presenting graphs and tables showing en- 
rollment, expenditures, and trends in instructional person^ 
nml in vocational education from 1970-1975. A discussion 
of accountability difficulties due to the lack of adequate 
data concludes with a recoiTmendation for a management in-- 
formation system that would detail to the public and to 
policy makers who is being trained and for what role, where 
the training is being given, how much it costs, and what the 
results aro. Data elements that should be included in such 
a reporting system are described and suggestions made 
include (1) the deletion of adult educatl' ' level of 
vocational education and the use of nrac: K-14 to 

replace iti (2) the omission of tradifci . ^pational 

service areas as data elements and use ©fined Office 

of Education code taxonomy of courses and ^ )b titl^^s tha^t 
will result in reporting data according to a taxonomy of 
programs and groups of job titles. 

Among the general observations resulting from Project Bass- 
line's five--year study of vocational education are the 
following z 



m Vocational education enrollments have incraased 
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m In proportion to tot^l nopulation, women and 
ethnic minority grout^^ h Uvfi a greater represent' 
tation in vocational uiu-mtiop than nther 
Begments of the y^jpulation. 

• ADproximately half of th- hi^n school popula- 
tion is reported as being srroUed in vocational 
e^^ucation. 

• Employment rates for vocational education 
proaram completers available for work is higher 
than for comparable age groups in the total 
labor force. 

• in comparison with atate and local support, the 
federal share of vocational education expendi-* 
turee has steadily declined. 

m The enrollment of manpower training programs 
under the U. S. Department of Labor is less 
than five per cent of the reported enrollment 
in vocational education. 

• There is a serious shortage of vocational edu-- 
cation tedch^rs. 

m Research has had an important impact on 

program development in vocational education 
since 1963. 

• While state and federal reporting has improved, 
there are still reporting problems that need 
to be solved, 

A eeries of Project Baseline SupplMental Reports published 
between 1974 and 1976 include the following titles i 

Appraisal of the Manpower Training Programs Establlshei? 

in the 1960s. Project Baseline Supple - 
^^^^^1 "odr^^r-^ n^nrft^ T.. Rrandnn . Flaastaf f I 
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Career Edu cation in the United States Today i What It 
Is^'WHere ^ and the RiBults So Far. Project B aseline 
Supplemental Report . Robert M.Worthington- FlagBtaff i 
Northern Ar~iEOna~ University, June 28, 1974, 55 pp, 
(ED 099 693) 

Impac t of VQcational Education and Hanpow er Training on 
I^rqeF Populations V Groups, the Disa dvantaged, 

Handicapped, Unemployed , and Unemployable A dults. 
ProjeGt Baseline Suppremental Report , James Wall, 
Flagstaffi ~ Northern Arizona University, July 31, 1974, 
54 pp. (ED 099 686) 

I mpact of Vocational Education and Manpower Training on 
the Labor^ Market, Project Baseline Supple mental Re- 
port . Gordon f\ Law. Flagstaff i Northern Arizona 
University, August 1, 1974, 52 pp. (ED 099 692) 

Impact of Vocational Education Research at the Federal 
ana~5tate Levels, PrQiect Baseline Supplemental Report , 
Robert MillerV Flagstaff i Northern Arizona University, 
October 25, 1974, 140 pp. (ED 099 687) 

Preparation of Teachers for Voaatlonal Education . 
Project Baselin e Supplemental Report, Gordon I, 
Swansoli. """Flagstaff I Northern Arizona University, 
September 1974, 34 pp. (ED 099 690) 

R eport to the Nation on Vocational Educati on, Project 
Baseline S u pplemental Report , nary L," Ellis. Flagstaff 
Northern^Ariiona University, November 197 5, 125 pp, 
(ED 116 052) 

Women in Vocational Educa_tion. proj ect Baseline Supple - 
mental Report. Marilyn Ste^^ae. Flagstaff^ Northern 
Ajcizona University. October 30, 1974^ 154 pp. 
(ED 099 609) 

Vocational Educati o n Planning in the State s i Pro j ect 
Base line^uppleme ntal Report / "fatrick j, ^O'Reilly. 
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National Longitudinal Study [of the High School Claes of 
~ 1972^ , Review and Annotations of Study Reports . 
Sainuer S/ Peng/ Celcille E. Stafford/" and Robin J* 
Tolbert^. Durham, North Carolina i Center for Educational 
Research and Evaluation, Research Triangle Institute, 
Research Triangle Park, Hay 197 7, 78 pp. 

The stated purpose of this publication was to "establi5«U an 
inventory of studies" that used the National Longitudln;:^! 
Study (NLS/72) of the High School Class of 1972 data be^e 
so that an exchange of ideas among usrsrs might take place 
and areas needing further investigation could be identified* 
Monitored and primarily funded bv the National Center for 
Education Statistics, the NLS/7 2 fo-usee on the educational, 
vocational I and personal develo; t^v^^rt of high school gradu-- 
ates and on the various person^-^^, 7?acial, and cultural 
factors contributing to that d^^W; - opment , 

The study began in the spring of 1972 with a national proba- 
bility sample of 19,136 seniors from 1,070 public, private, 
and church-affiliated high schools. Added to the base-year 
sample v;ere another 4,315 students of the class of 1972 from 
24 8 schools that had been unable to participate in the 
earlier survey. Each student was asked to complete a stu-- 
dent questionnaire and to take a 6 9--minute test battery. 
Survey administrators were asked to complete a student 
record form and a school questionnaire, and school counselors 
were asked to complete a special questionnaire designed to 
provide data about their training and experience. 

The first follow-up began in October 1^73, the second in 
October 1974, and the third in October 1976, The data 
files for the base year and the first and second follow--up 
surveys have been merged and have been used by researchers 
to investigate various educational, vocational, and social 
issues* 

Chapter 2 describes the NLS/72 data base and data coverage 
in terms of content areas and time frame in which variables 
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m Half of the black studentvi^ per aent of 

the white students were Gnro^ivjd in general 
programs , 

m A quarter of the black students and about 20 
per cent of the v;hite studeiits were enrolled 
in vocational programs . 

m Both vocational and general students had lower 
scores than academic students on the acadeinie 
ability tests, 

• Compared to general education s*:udents^ voca^ 

t tonal education students achieved higher grades ^ 
were less alienated toward school^ and were more 
focused in their post=secondary work aspirations. 

• Job training received in hicrh school corr:^iated 
positively with occupational attainment and 
income after graduation. Participants were 
more likely to be employed and had a higher 
median inao* than non--participants , 

An annotated bibliography of over 150 documents pertaining 
to the NLS/72 data is presented in Chapter 4, with names 
and addresses of data file users appearing in the index. 
Following is a partial listing of NLS/72 publications: 

Changes in Att itudes One and One-Half Years After 
Graduation > William B, Fetters* Washington/ D.C.i 
National~Center for Education Statistics, 1974^ 14 
pp. (ED 117 155) 

National Longi tudinal Study of Hich School Seniors , 
An^qendci for Vol ic y Research. Stephen J. Carroll and 
Peter Aritorrison. Santa Honicas Rand Corporation ^ 
June 1976, 80 pp. (ED 135 818) 

National Longitudinal Study of High Schoo l Seniors , 
Fulfillment of Shor^ Educational Plans and 

n^i-ii-T nii^ni-*© 1 n"1p^^^^ William Fetters and 



National Longitudinal Study of the High School Clais of 
1972. Capsule Description of Firat Follow-up Survey 
Data. bV K* Eckland and j, p. Bailey* ReeearGh Triangle 
Institute. Durham, North Carolina i Research Triangle 
Institute, Center for Educational Resaarch and Evalua- 
tion, 1976, 47 pp. (ED 120 251) 

National Longitudinal Study of the High School Class of 
1972"/ Capsule Description of School Seniors . 

Base*-Year Survey . Wiliiam B. Fetters. Washington, D.C. i 
'National Center for Educational Statistiae, 1974, 43 pp. 
(ED 097 368) 

National Longitudinal Study of the H igh School Class of 
1972 t Capsule Deacrlption of Second Fq11qw-up Survey 
Data. October 1974 . Bruce K, Eckland and J, P. Bailey, 
Jr. Durham, North Carolina i Research Triangle Insti^ 
tute, Center for Educational Evaluation and Research, 
1977, 62 pp, (ED 139 826) 

National Longitudinal Study of the Hi^ gh School Class of 
1972 . Comparative Prof iles One and 'ci ne-Half Years After 
G raduationr r William B- Fetters. Niitional Center for 
Education Statistics, 1975, 68 pp. (ED 120 202) 

National Longitudinal Study of the f Aigh School Class of 
1972 . CounBeior" Questionnaire . Educational Testing 
Service - Princeton i " EduGational Testing Service , 1972 , 
8 pp. (ED 097 371) 

National T.ongitudinal Study of the High School Class of 
1972. ylr st Follow-up Survey Design, Instrument Pre - 
p^h tini. r Data Collection , and File Developttient ^ 
KerlKetK™abl^, Washington, D.C. i~ National Center for 
Education Studies, 1977, 48 pp. (ED 141 406) 

National Longitudinal Study of the High School Class of 
1972. Parent's Questionnaire . Educational Testing 
Ser\n ce. Princeton s E'ducational Testi.ng Service, 
19^3, 7 pp. (ED 097 370) 
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National L o ngitudinal Study of the Hig h School Class of 
1972 V St u tent Questionnair e > Educational Testing 
Service, '^^ncetons Educational Testing Service^ 
1972, 26 pp. (ED 097 369) 

National Longitudinal Study of the High School Class of 
1972. S tudent Questionnaire and Test Results by Sex ^ 
High Sc hool Program, Ethnic CategoryV "and Father's 
Education, William B.' Fetters , WashingtonT D.C, i" 
National Center for Education Statistics, 1975, 127 
pp. (ED 111 850) 

National Longitudinal Study o f the Hi gh School Class of 
IjJTj , "Student ' s "School Record Infformation . Education 
Testing" Service . Princeton: Educational Testing 
services, 1972, 6 pp, il^ 372) 

National Longltudl n^'; ^ Stuo ^ the Hicrh School C lass of 
1972 p SympQslumT ids in Po rjt-secondary Education . 

J. P. Bailey, ~Jr, , ^a^. Waehington, D.C, t American 
Educational Research Aisociatlon , /^prll 1977, 100 pp, 
(ED 142 126) 

National Longitudinal St u dy of the High S cho ol Class of 
1972. Tabular Summary of t he rirnt F ollow-up'' Q uestion - 
haire Data > Kehneth Tabler, ^urliam, North Caroiina^ 
Research Triangle Institute, 1C7C. 965 pp. 
(ED 143 677) 

Nation al LoAigitudlnal Study of the High Scauol Class o f 
1972 / Tabular "Sunmiary of Student nueBtionnarie DataV 
^oIumgll> (Quest ion a 1-37) i Bruce Thompson, 
princetoni EHucational Testinr^ Service, 1974, 459 pp* 
(ED 103 452) 

Nation al Longitudinal S tu dy of the High flohool Class of 
1972 / " Tabular Summary of^ Student Questionn aire Data . ' 
Volume 2 . "(Questi ons^ 30-95 ) . Pr uce VI , Thompson . 
Princetons Educational Testing Service^ 1974, 475 pp, 
(ED 103 453) 



ances, it was acknov/Ledged that both dropping out and 
unfavorable labor Tnarket eKperienGeB may be attributed to 
social and p^rscnal disadvantages* 

Following are some of the differences found between graduate 
and dropouts t 

m Graduates had lower rates of unemployinent 

and were more likely to be employed full time* 

• Graduates had advanced to higher occupational 
levels. 

• Graduates were more likely to have participated 
in a formal occupational training program* 

• While inmediats monetary advantages of complet-^ 
ing high school are not evident, by the third 
year after leaving school , graduates were earn-- 
ing higher hourly rates of pay than dropouts* 



those who are unionized experience iess unempiuynienu uiion 
those who are non-unionized. 



* * * 



Career Thresholds; A Longitudinal Study of the Educational 
and Labor Market Experience 5 of Young Man . Volume 6 . 
Andrew I. Kohen, John T. Grasso, Steven C. Myers, and 
Patricia H. Shields. Columbus: Center for Human 
Resource Research, The Ohio State University, March 1977, 
383 pp. (ED 138 831) 

This volume analyzes the educational and labor market expe- 
riences from 1966 to 1971~a five-year period which could 
be expected to reflect not only the impact of the Civil Righti 
movement and the effects of a period of relatively high 
employment (1966 to 1969) followed by a period when unem- 
ployment beoan to rise (1969 to 1971) but also the impact 
of military' service on subsequent labor market experience 
for those returning to the civilian labor market. 
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November iy73, 4yy pp. 



Volume 2 is charaaterized as a companion volume to the first 
in the series of Project Baaoline reports, 3ince the first 
and second volumes were issued simultaneously^ many of the 
tables in the second volume contain first-year data not 
found in Volume 1. Vocational education enrollment ^ cost, 
and placement data for the 1971=1972 school year are ra-^ 
ported according to secondary/ post- secondary ^ and adult 
levels of instruction by program areas. 

Serious limitations to the usefulness of the date are eK- 
plained as outgrowths of problems of definition^ communica-- 
tion^ and, in some cases? mishandling of data collected by 
state and federal agencie , 

* * * 
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'■ - ■ / 

Congress should include in the legi.s l«t ioii _ 
for 1975 a set of vocational education defi- 
nitions, allowing for fleKibility and, at the 
same time, providing for national standardiza- 
tion of terminology. 

A new reporting system should be established 
for both vocational education and manpower 
training in which data elements in machine 
readable form can be transmitted from local _ to 
state to federal level for computer tabulation 
and analysis at each level. 

* * * 
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sisting probleiTiB resulting from the lack of a nationally 
standardised set of definitions and inconsiatant handling 
of data are pointed out, " ^aviaw of information eystems 
currently in use in Gach v .te is included along v/ith a 
report of a special task iorce effort to bring uniformity 
to definitions of k^jy terms used in raporting data, include 
ing the term "vocational education*" 
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tplace it I (2) the omission of tradifci ^ ^patxonai 

irvice areas as data elements and use ©fined Office 

' Education code taxonomy of courses and ^ ib titl^^s thi^t 
.11 result in reporting data according to a taxonomy of 
•ograms and groups of job titles. 

long the general observations resulting from Project Base- 
ne's five-year study of vocational education are the 
allowing z 

m Vocational education enrollments have incraased 
nationally. 

m Vocational education has not eKpanded for the 
disadvantaged and handicapped at the same rate 
as for other students^ 

• Cooperative and work study programs have 
reached only a fraction of their potential 
enrollment. 

• Performance of the statee varies widely on 
all measurements. 
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le state and federal reporting has improved, 
ire are still reporting problams that need 
be solved* 

' Project Baseline SupplMental Reports published 
'4 and 1976 include the following titles i 

. sal of the Manpower Training Programs Establlshei? 
^ csB in the 1960s, Prolect Baseline Supple - 
/ Report *" "George L. Brandon/ Flagstaff i 
?rn Arizona University, August 30, 1974/ 97 pp* 
19 691) 

l ing Vocational Eduoation Hesearch and Development * 
?t BaB~el3,ne Suppremental Report * Commxttee on 
.onaT Education "Research and Davelopment , 
Lgton, D.C*i National Acadamy of Sciences, 1976, 
>. (ED 128 654) 
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iisiline l ^upplemental Report , jlary L. Ellis. riagstaEr 
Northern Arizona University, November 197 5, 125 pp. 
(ED 116 052) 

Women in Vocational Educa/tion. Pro ject Baseline Supple - 
mental Report. Marilyn Ste'^ae. Flagstaff; Northern 
Arizona University. October 30, 1974 , 154 pp. 
(ED 099 609) 

Vocational Educati o n Planning in the State s; Project 
Baselln i^uppleme ntal Report . ' fatrick J. O'Reilly, 
flagstaff ! Nor thern Arizona University, December 
1975, 60 pp. 



* * * 
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The first follow-up began in. Octc 
October 1974, and the third in Oc 
files for the base yaar and the i 
surveys have been merged and hav€ 
to investigate various educationi 
issues * 

Chapter 2 describes the NLS/72 da 
in terms of content areas and tin 
were measured. Findings from the 
NLS/72 data are sumnarized in Che 
following I 

• The HLS/72 sample repres€ 
high school senior s in 1£ 
cent white, 9*5 per cent 
from other minority grouE 

• A quarter of the black st 
of the white students wei 
programs * 
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National" Center f 
pp. (ED 117 15£ 



Nationa l Longltuc 
An Age nda Cor VoJ 
Peter A. Itorrisor 
June 1976, 80 pp. 

National Longituc 
Fulfillment o f Sl^ 
Continuance in Ec 
others . Washing^ 
tion Statistics > 

National Longituc 
1972 / Base- Year 
Results by Acadeii 
Region * William 
National Center 1 
(ED 120 252) 
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Seniors * 
Carroll and 
rporation e 



Sen lor a » 
i and 
ire "and 
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iool Claas of. 
lire and Test 
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L976, 106 pp. 
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National Longitudinal Surveys Handboo k. Herbert S. Parnea. 

Columbus! >lenter for Human Resource "eaaarch, The Ohio 

State Univeraity, Revised November l*^??, 87 pp. 

Since 1966, national Longitudinal Surveys (ULS) have been 
conducted to fiather data on the labor market experience 
and other eharncteriatics of men 45-59 years old and young 
men 14-24. In 1967 a third group, TOnen 30-44 , was added,- 
and in 1968 a fourth group, young vromen 14-24, was added to 
the study. C?he NLS project, sponsored by the U.S. Departrant 
of Labor, is carried out by The Ohio Ptate University's 
Center for Ku; n,n Resource Research (Dr. Herbert Parnes, 
Projeet Director) . Studies utilizing ITLS data (see list 
of titles belov;) attempt to identify characteristics that 
appear to be most important in explaining variations in _ 
employinent , unenployment , and labor force mobility. National 
probability saniples of approximately 5, 000 were drawn to 
represent each group; and personal Interviews in addition to 
niail and telephone surveys have been made periodically. 
Because of the uideepread interest in the data and low at-- 
trition rates uithin the groups, the decision was made to 
increase the ejctent of the surveys, from the oriqinal period 
of 5 years, to 15 years. Final interviews are scheduled in 
1981 for the older and younger men, 1982 for the older women, 
and 1983 for the young women. 

Volumes reporting findings from surveys already conducted 
are cited in Category 1 under the following titles: 



Career Thresholds: h Longi tudinal Study of the Educa - 
tibnal and Labor Market Experience of Male Youth. 
Volumes 1-6 . 

Dual Carciars: A Longitudinal Study of Labor Market 
Experienee of Women. Volumes 1-4. 

Pre-Retirement Years; Five Xears in the Work Lives 
of Middle- Aged Man. Voluines 1-4^ 

Years for Decision: A Longitudinal Study of the EJ.U " 
catlonal and Labor Market Experience of Young Women . 
Volumes 1-4 . 

Surveys scheduled for a newly developea panel of young men 
and young wonen (v;ith the first survey scheduled for 
January 1979) are expected to enable replication of analy- 
ses made of earlier cohorts and will help evaluate the 
expanded employnent and training programs of the late 1970s. 
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The Handbook describes NLS sampling, interviewing, and esti- 
mati m proced ures and provides a table of survey schediiles 
and a listing of variables investigated in the surveys for 
each of the groups. Characteristics of the data files are 
aescribed, and information is provided on how to obtain 
data tapes. A National Longitudinal Surveys Bibliography 
of approximately 300 entries provides a list of the compre- 
hensi^?e and special reports generated as a result of the 



surveys 



* * * 

* * * 



The Pre-Retirgment_YM£gj--^-LQ ^ Study of the Labor 

yolume 1. He r bert S. Parn¥s; Belton M. Fleisher , Robert 
C M ilTiI s , and Ruth S. SpitE. Columbus: Center tor 
Human Resource Research, The Ohio State University, 
October 1968, 303 pp. (ED 026 525) 

This report is the first in a study of labor market escper- 
ience and behavior of men who were 4 5 to 59 years of age in 
1966 Data gathered from a national probability sample ot 
5,000 men relative to economic, social, and P?y°^°l°5i=al 
variables are analyzed in an attempt to describe and explain 
differences in labor force participation, commitinent to 
work, propensity to retire, rates of pay, and 30b satisfac- 
tion . 

Other volumes reporting survey data from the same sample 
through 1971 are these: 

The Pre-Reti rement Years_i__ A Longitudinal S tudy of /^he 
^^Srm^tBKpmE^^^^^^^^B^^^- Herbert S. 
^^^T:^-Bmiwr-Tol^^r~^^Et^rT5^ Human Resource 
Research, The Ohio State University, January 1970, 
122 pp. (BD 039 331) 

The Pre-Re tirement y -^^i-°- a Lnnoitudinal Study of the 
t^b^^ Ma^k^4-^xpgrience ^05"- Volu"^e A - Herloert S. 
^^5— i-TBthitir'^^^ Human Resource 

Research, The Ohio State University, August 1972, ibi pp. 
(ED 068 682) 
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The Pge-RetiremerLt Years s Five Years in the Work Liv^ea of 
Mrddla^Aged Men^ Volume 4 . Herbert S. Parnes and others, 
Columbus: Center for Human Resource Research, a^ha Ohio 
State University, December 1974, 425 pp. (ED 103 052) 

Additional surveys of the men in this group were conducted in 
1973, 1975i 1976, and 1978 and are Bcheduled for 1980 and 1981. 



i£ it 
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Projeet Talent. The American High Sohool Student . John 
Cr Flanagan, Frederick' B. Davia, John T* bailey, Marion 
P, Shaycoft, David B- Orr, Isadore Goldberg, and Clinton 

Neyman, Jr. Pittaburghi University of Pittsburgh, 
1964, 738 pp. 



One of the major publications in a series of Project TALENT 
reports published since 1960, this volume describes the 
effort to determine the aptitudes and abilities of the 
iiation's high school students. In 196 0 a probability sample 
Qf over 440,000 students in the 9th , 10th, 11th, and 12th 
graces in 987 randomly selected high schools across the 
country were administered a comprehensive set of educational 
and psychological testa and inventories* The more than 
2,000 items included were intended to produce information 
that would describe talents, help to establish standards, 
and lead to identif ieation of aptitude and ability patterns 
predictive of success in various careers. 

The folio v^-- up surveys scheduled one, five, ten, and twenty 
years following the year of g-raduation for each class were 
planned to trace the careers of the students in a continu-^ 
dng effort to maintain an updated data bank on the original 
sample. 

Prev^iously published Project TALENT publications provide 
detailed information on the Project's basic plans, the 
instruments used in the survey, and the characterlstice of 
American high schools- 
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Project Talent Data B anki A Handbook. David V, Tiedeman. 
Palo Alto, California I American institutes for ReRearchi 
April 1972, 119 pp. 

This publication is intended to provldG researchere inter-^ 
ested in the Project TALENT Data Bank an overview of the 
Project's purpose, measures, sampling, weighting, and 
follow-up procedurei in addition to information about the 
special data files available for further study* Services 
available from the IiiBtitute include printouts, worktapes^ 
and Gontract research* Hypothetical examples of the kinds 
of prDjects made possible through the use of Project TALENT 
data, as well as actual projects which have used the data 
bank between 1964 and 1971 rare sunmiari^ed in the appendix 
to the Handbook . A list of Project TALENT publications 
through 1971 is also provided. 

Project Talent One-Year Follov/-up Studies . John C, Flanagan 
and William W. Cooley. Pittsburgh i UniverBity of 
Pittsburgh, 1966, 387 pp. (ED 101 087) 

This report describea the follow-up procedures used and the 
data gathered fron mail surveys of the original Project 
TALENT sample of over 440,000 high school students under-- 
taken one year following graduation of each of the high 
school classes from 1961 to 1964. Data on the schools they 
attended, the jobs they held, the careers they planned, and 
choices they made are reported and related to information 
collected from the 1960 survey in an effort to determine 
how young people choose careers and hov/ educational exper-- 
iences prepare them for those careers. 

The final chapter describes the use of a computer measure- 
ment system by guidance programs to maintain cumulative^ 
and progress records, to evaluate educational productivity 
of curriculum and staff, to project vocational potentials 
for students, to monitor and prescribe individual learning, 
and to manage continuation services to graduates and school- 
leavers* The reference section includes a listing o£ 
Project TALENT publications, published articles, and^ re= 
search reports delivered by staff members at professional 
meetings , 
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Years fo r Declslo m A Longitudinal Study of the Eduoatlonal 

John RT Shea, Roger DV^^W Frederick A. Zelier,* 

and Anflrew I. Kohen. Columbus i Canter for Human Resource 
Research, The Ohio State UniverBity, February 1971, 245 
pp. (ED 049 376) 

Data gathered in the initial 1968 National Longitudinal 
Survey of a representati^fe sample of 5,000 young woman 14 
to 24 years of age is analysed and reported in this first 
of four volumes published under the title Y ears for DeGlsion , 
Marital and familial characteristics, financial status, 
education arid training, health condition, attitudes and the 
labor markat are exaniined as eKplanatory variables and then 
related to labor force participation, unemployment, mobility, 
and educational and occupational aBpirations of the young 
women ^ 

Tne data shov; that the yeari between 14 and 24 are critiQal 
for young TOinen faced with pressures and options i whether 
to continue or to leave school, whether or not to gat 
married, whether to stay with parental family or to develop 
independence. While these decisions are recognized to be 
interrelated with decieione about education and work, 
causal relationships are unclear* 

Findings from the initial survey include the following i 

• Approjcimately 70 per cent of the young women 
14 to 17 years of age enrolled in school say 
they would like two or more years of post-- 
secondary schooling* 

m The highest level of school completed is re- 
lated to the occupational assignments and the 
hourly rate of pay of the young women who are 
employed* 

• Shorthand and/or typing skills acquired in 
high school and occupational training outside 
the formal gchool are related to employinent 
generally and white-collar employment ipeci^ 
iically p 

• Marital and family status have a direct in^ 
fluence on a woman's occupational assignment 
and is interrelated with such factors as 
educational attainment and health, 

* * * 
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Years for DeclBion; h Longitudinal Study of the Eduaational 
and LaborMarket Experience of Youn^ Women ^ Volume 2 , 
Roger DV Roderick'and Joseph M. Davis/ Colunibusi Center 
for Human Rasource Research^ The Ohin State University, 
March 1973, 144 pp. (ED 076 812) 



This volume investigates the eKtent to which changes oc - 
curred in the educational and labor rnarket status and aspira 
tions of young women during the year following the original 
interview* The following findings resulted froin the 1969 
interviews t 

m The dropout rate for young white women vas 
comparable to that shown for young white men 
between 1965 and 1976 (MLS, Career Thre sholds ), 
but the dropout rate for young talack women at 
the high school level exceeded that of their male 
counterparts , 

m The percentage of young women revising their 
educational goals downward was similar to that 
of young men; however, in contrast to the 
young mmn, who reported economic factors as the 
reason for revising goals r young women most 
often cited changes in '^interest." 

m For young women out of school on both survey 
dates who had completed exactly 12 years of 
schools those who had been enrolled in voca- 
tional or commercial programs made the greatest 
gains in rate of pay during the year, while those 
from general curricula made the smallest, 

• Job changes were related to higher hourly earnlngB 
and increased job satisfaction. 

• hs measured by the occupational information 
test used in the survey / white young women have 
substantially more labor market knowledge than 
black young women---a variation that persists 
even after controlling for educational attain- 
ment and mental ability* 



1^ # * 
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Years for Pealslons A Longitudinal Study of the Educational 
and Labor Market EKperiBnce "of YQung Women, Volume 3 . 
Roger D. Roderick and Andrew I. Kohen. Columbus i Center 
for Human Reaource Research^ The Ohio State University , 
Decarr^er 1973 , 111 pp. (ED 094 135) 



The third volume in the series of reports examining labor 
market and aduGational eKperiences and aspirations of young 
women includes data gathered in the 197 0 interviewa with the 
young woman 14 to 24 years of age at the time of the initial 
survey in 1968, Included in the data were the following 
findings i 

m The young women continued to revise their 

goals, bringing thei^ more in line with "reality." 

m Expected levels of educational attainment 
dropped slightly. 

• Forty per cent of the young women had changed 
their plans for age 35^ overwhelmingly in the 
direction of employment at that age* 

m ApproKimately 20 per cent of those who had never 
been married in 1968 were married in 1970, and 
more than 10 per cent who had been childless 
in 1968 had at least one child by 1970* 

• Inter firm changes in employment were related 
to change in occupation/ increase in hourly 
wage^ and job satisfactionp 

* * * 

Years for Decision; A Longitudinal Study of the Educational 
and Labor Market Exper lance of young Women/ Volume 4 * 
Prank L. Mott, Steven H* Salidell , David Shapiro, Patricia 
K. Brito, Timothy j. Carr, Rex C. Johnson, Carol L, 
Jusenius, Peter J* Koenig^ and Sylvia P. Moore* 
Coluinbusi Center for Human Resource Research, The Ohio 
State University, November 1977, 366 pp, (ED 147 540) 

The research studies in this volune indicate that a substan- 
tial portion of young women / represented by those who were 
14 to 24 years of age in 1968, will spend a major part of 
their lives in the labor force. Most women continue to 
plan careers in traditionally female positions and as a 
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result continue to occupy lower paying positions than those 
occupied by men. 

Findings of the six studies reported in this volume include 
the following; 

• Parental factors directly affect a young woman ' s 
educational experiences and have a significant 
effect on the types of career path she plana to 
follow. 

• A young biack woman with socioeconomic back- 
around characteristics similar to the average 
white young woman is more likely to have col- 
lege aspirations and is also more likely to 
attend college than her white counterpart. 

9 There is evidence that women committed to the 
labor force are willing to accept low Initial 
wages as a price for job training promising 
higher lifetime earnings. 

• Marriaae disruption increases the likelihood 
that young v/omen will enter the labor force. 

• Non-participation in the labor force associated 
with childbirth increasingly tends to be brief. 



* * * 

* * * 



You th in Transition. Volu me 1. Blueprint for a Longitu- 
dinar~Study of Adolescent BoyiT Jerald G. fiachman , 
Robert L.Kahn, Martha tT Mednick, Terrence N. Davidson, 
Lloyd D. Johnston. Ann Arbor: Institute for Social _ 
Research, University of Michigan, 1967, 274 pp. (ED 026 46. 

This first in a series of six volumes summarizes the plans 
for the siK-year longitudinal study of a national sample 
of approximately 2,200 young men in 87 high schools. The 
first data collection in the fall of 1966 was made when 
the boys were in the lOth grade. Follow-up surveys were 
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conduoted in the spring of 1968^ the spring of 1969, and in 
June and July 1970 when most of the boys had been out of 
high school for about a year. Volu«e 1 contains deecrip- 
tions of study design ^ sampling procBdures, response rates, 
and questionnaires. 

Following are titles of other volumes in the series: 

Youth in Transition, Volume 2. The Impact of Family 
Background and Interiigence on Tenth-Grade Boys , 
Jerald cr Bachman. Ann Arbori Institute for Social 
Research, 1970, 293 pp, (ED 051 414) 

Youth in Transition p Volume 3 . Drop ping Out-'^Problem 
or Symptom ? Jerald C, Bachnian and others , Ann Arbor: 
Institute for Social Research^ University of Michigan, 
1971, 255 pp. (ED 059 333) 

Youth in Transition, Volume 4, Ev^olution of A 
^rateg y for Lon gitudiHal AnalyBis of Survey Panel Data ^ 
Terrence M* Davidson, Ann Arbori Institute" for Social 
Research, University of Michigan, 1972; 130 pp. 
(ED 064 440) 

Youth in Transition. Volume 5* You n g Men and Jlilitary 
Service , Jerome Johnston and Jerald C> Bachjman* 
Ann Arbor: Institute for Social Research, university 
of Michigan, 1972^ 259 pp* (ED 072 185) 

Youth in Transition. Volume 6, Adolescence to Adult - 
hood! Change and Stability in~the Li ve s of Young Min . 
iJerald C- Baahman and others, Ann Arbor: institute 
for Social Research, UniversitY of Michigan, 1978^ 
351 pp. ^ 



* * * 
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Category 2: REVIBW AND SYNTHESIS PAPERS 
LIST OF TITLES^ 



Program Evaluation In Vocational Educations A Review , 
Floyd McKinney / The Center for Vocational Educa- 
tion^ The Ohio 'state University, 1977. 



Page 



Evaluation of Vocational Techn ical Education, Phase I . 
New Educational Directions, Inc. Indianapolis; 
The Indiana State Board of Vocational Technical 
EdUQation, July 1975^ 25 pp. 



^complete bibliographic citations for these titles, 
along with supplementary information are provided with 
annotations on the pages that follow - 
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Review and S ynthesis of Cost-Eff ectivenees Studies of 195 
~ " Vocational "and Technical Education , Ernst W, 

Stromsd'orfer , The Center "for Vocational Educa- 
tion, The Ohio State University, 1972. 

Review and Synthesis of Resgarch on Cooperative Voca - 197 
" " tlonal Education . Harold R/ Wallace, The Center 
for Vocational and Technical Education^ The Ohio 
State University, June 1970- 

Review and S ynthesis of Research on the Economics oC 199 
VoBSt^nkl Edu cation . J. Robert Warmbrod , The 
Center for Vocational and Technical Education^ 
The Ohio State University, November 1968. 

Review and Synthesis of Resea rch on the Placement and 
" Follow-up of Vocational Education Students , 

J . Kenneth Little "The Center for Vocational and 
Technical Education, The Ohio State University, 
February 1970, 

A Synthesis o f Research Findings Which Describe Se - ^ 201 
lected "Benefits and Qutcomes for Participants in 
Vocational Education, Doug Sparks. Bureau of 
Occupational and Adult Education. U.S. Department 
of Health, Education, and Welfare, October 1977. 

What Hap pens After Training? A Review of Follow-Up 203 
of Vocational Gradu a^tes. Krishan K. Paul. 
Urban Observatory of Metropolitan Nashville and 
University Centers, September 30, 1976. 



2O0 



192 



Evaluation of Vocational TechnlQal Eduaatlon. Phase I . 
New EdUGational DireGtionS/ Inc. Inaianapolls s The 
Indiana State Board of Vocational Technical Education, 
July 1975, 25 pp. 



Using responses from State Departments of Public instruation 
and information resulting from an ERIC search, this document 
reports on the statue of evaluation activities by the var- 
ious states according to these categories i self-study, 
visiting team, follow-up of students, employer and/or com- 
munity evaluation, cost/benefit analysis, behavioral 
objectives, non-^behavioral objectives, and job-placement 
relatedness. 

The report concludes that regardless of the method^ or 
combination of methods, used for evaluating a particular 
school or program, it Ib of primary concern that the eval- 
uation system meet the needs of the school or program in 
that it results in a cooperative effort to iniplement the 
findings* A bibliography of 2 61 aitations is included. 



Program Evaluation in Vocational Educ ation; A Review . 

Infornmtion Series No. 117, Floyd HcKinney. Columbus n 
The Center for Vocational education. The Ohio State Uni 
vereitv, 1977, 41 pp. (33D 149 186) 



The author points out that even though it is generally 
agreed that the evaluation of vocational education prog^fr.no 
is necessary for better declBion^makinr^ for various reason 
a comprehensive and systematic approrncli to program evalua^ 
tion has been slow to develop. 

In this review; of literature on evaluation of vocational 
education since 1970^ reference is nar'^Q to several concep- 
tual schemes and models that have been used typically in 
program evaluation by researchers and education agencies. 

Since it is reasoned that identifying needs-'^aB may be 
reflected in program objectives' — io basic to an evaluation 
affortf a discussion of several needs«asseBsment surveys 
is included; providing suggestions for sampling techniques 
and questionnaire content. Summarized strategies for con- 
ducting former student follow-'Ups, employer surveys^ 
student-^parent surveys, and cost=benefit studies contain^ 
procedural checklists and suggestions for those considering 
these kinds of studies. 

In a discussion of management inforTPation systems, specific 
reference is made to the systems that have been developed 
for Florida, Massachusetts, and Tenneonee* 

Studies carried out by the Kentucky, South Carolina, 
Oklahoma, Washington, and Florida State Advisory Councils 
are summarized; and it is noted that while most state ad-- 
visory councils use existing data in the preparation of 
their annual reports, several have also initiated special 
research and cost=benefit analysis studies. The U.S. 
Office of Education, the American Vocational Association, 
the departments of vocational education in each state, and 
various national, regional, and professional accrediting 
agencies are cited as additional sources of standards, 
sainple questionnaires, and guidelines for use in measurinr 
and assessing program characteristics and other aspects of 
vocational education . 

Based on his review of the literature and eKperience in 
vocational education program evaluation^ the author has 
arrived at the following conclusions; 
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• While "one^shot" evaluatiori efforts in voaational 
edacation have some value, program evaluation 
should be a Gomprehensiva and continuous effort* 

• While more eitiphaais should be placed on cost 
analysis studies/ it shoulcl be realiEed that such 
studies leave out important subjective factors 
that avery deGiaion- maker must consider, 

• Institutions and agencies need to devote more 
resources to developing program objectives since 
measurable objectives are esGential to program 
evaluation. 

• Research in all areas of methodologry is needed. 

m Since prograni evaluation depends on a continuous 
flov; of reliable and valid information/ system-- 
atic and comprehensive information collection 
is neadad. 



Review and Synthesis of Cost--Ef f ectivaness Studies of 

Vocatibiial and Teohnlcal Education . Information Series 
No, 57, "Ernst W. Stromsdorfer * Columbus i The Center 
for Vocational and Technical Education^ The Ohio State 
University, 1972, 151 pp. (ED 066 554) 



The author^ of this study identifies the non-economist as 
his intended audience and begins his study with an over^ 
view of the use of economic analysis and theory which is 
fujidainental to an understanding of cost-benefit analysis 
studiea. Inaluded in this section of the report is a dis^-^ 
Gusaion of optimum allocation of public eKpendltures for 
vOGational education through an appropriate investment^ 
decision rule which considers net present value^ internal 
rate of return^ and benef it--cost ratio* 



* * * 
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Soma of the conceptual and measurament problems that need 
to be addraasad in the analysis of educational investment 
ara pointed out i earnings versus utility maKimization ^ 
non-market production and consumption, impact of education 
on values and preferences^ complementarity and inseparabil 
ity^ of human skills, external ef facte, influence of unem^ 
ployment on the determination of costs and benefito, and 
the problems of the control group. 

Drawn primarily from studies, papers, reports^ and monographs 
published in the 1960s that attempt to evaluate vocational 
and manpowar training programs, the report states that 
ralativaly few i^ere found that dealt adequately with the 
invastmant aspects of the programs. Problems with inter- 
preting the implications of cost-'benef it studies evolve 
from v;eaknasses in methodology such as lack of control or 
appropriata comparison groups, inadequate sampling and non- 
response procedures^ failure to deal with self-selection 
bias, failure to use sophisticated procedures for the 
analysis off data, and failure to establish^ beforehand, 
acceptable levels of statistical significance- 
Following are soma of the conclusions draw by the author 
as a result of his reviews 

Secondary Vocational Education : Based on evidence from 
studies, saoondary vocational curriculum seems to yield 
greater labor market benefits than the comprehensive cur-- 
riculum- Consideration needs to be given, however, to the 
dissimilarity of objectives and populations served by the 
curricula. Cost--benef it measures which compare sets of 
vocational skills are lacking, and information on gross 
earnings or wage differentials by skills loses its useful- 
ness in the absence of cost information* 

Post-Secondary Vocational Education and Junior CQlleg_ej 
For students~who~ already have a" sound "high school voca- 
tional preparation, further post^secondary vocational ed- 
ucation does not yield substantial benefits. While greater 
benefits are associated with graduation from junior college 
than from secondary vocational education training, the 
existence of objective and population differences need to 
be taken into consideration when the data are interpreted. 

Manpower training : Results of studies of training programs 
in this category indicate they ara a worthwhile investment 
but causal factors have not been isolated. Manpower train- 
ing is viewed as a necessary complement to vocational 
education training when consideration is given to the rapid 
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changes in technology and the differences in the populations 
served by the two programs. 

The report suggests that in spite of limitations, cost^ 
benefit studies provide a rational, systematic approach to 
the economic analysis of programs designed to improve 
efficiency of the labor force and to increase the overall 
welfare and well-being of society* However, costs of well-^ 
designed, comprehensive studies will continue to limit the 
widespread use of this form of prograiti evaluation. 



* * * 

* * * 



Review an d Synthesis of Research on Cooperative Vocational 
~ Education / Research Series' No * 60. Harold R. Wallace. 
Columbus i The Center for Vocational and Technical 
Education, The Ohio State University, June 1970, 124 
pp. (ED 040 274) 

The research topics covered by this review of studies and 
literature published between 1962 and 1969 are arranged 
into four categories with a corresponding bibliography 
topically arranged. There are four focus areas i 

• The Student Learner i This section includes a 
re view of research concerned %vith self -concept , 
work adjustment, and special needs-^^data that 
may be considered as input for curriculum 
development and implementation in cooperative 
vocational education* 

^ The Employment Community t Manpower analysis 
studies, manpower" data sources, task analysis 
models, and occupational analysis studies are 
reviewed in this section, 

• Educational Technology i Procedures, products, 
and problems of research dealing with instruct 
tional objectives and content are discussed, 
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and a review o£ selected research concerned 
with teaching techniques and ^-^roc^^clures is 
included, 

9 Prograin Irnplementation i This section reviews 
research and literature dealing with program 
administration and evaluation. 

Few of the studies reported drainatic evidence of the value 
of one inetructional technique or one instructional syeten 
over another. Reasons identified wera the lack of pre- 
test/post^test instruments, techniqueo mC procedures 
sensitive to measuring what should be neaoured, and inade-- 
quate design of research studies that contirol for inter- 
vening variables. The author reports that the most suc- 
cessful studies investigated a single factor and provided 
obviously similar situations for the eKperimental and 
control subjects. 

The author concludes that the basic concepts of oooperative 
vocational education need to be logically organized, arti-- 
culated, and subjected to systeiTiatic theory building, 
leading to the establishment of general guidelines. 
Research needs and priority areas identified by the author 
include the following i 

• A theoretical and philosophical framework for 
research and development in vocational 
education 

• Interpretation dissemination oyctems for 
manpov;er data to be used by program 
developers 

• Research technology to provide for the trans- 
formation of occupational analysis data into 
instructional objectives 

• Techniques and instruments for the evaluation 
of instruction 

• Study of the contribution of occupational 
experiences to professional training and the 
impact of teacher-coordinator characteristics 
on the behavior of the student-^learner 

• Research involving psychological constructs 
such as "work values" and "personal adjustment" 

• Dissemination of research and dovelopment products 

* * * 

* * * 
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changes In technology and the differences in the populations 
mmtVBd by the two programs * 

The jreport suggests that in spite of limitationB , cost- 
benefit studies provide a rational, systematic approach to 
the economic analyaiB of programs designed to improve 
efficiency of the labor force and to increase the overall 
welfare and :?;all-taelng of society.* However, costs of well- 
desigiied, comprehMsive studies will continue to limit the 
widespread use of this form of program evaluation^ 



* * * 

* * * 



Reviev and Synthesis of Research on Cooperative Vocational 
Edviaation / Research Series No. 60. Ifarold R. Wallace, 
Columbus t The Center for Vocational and •Technical 
Education, The Ohio State University i June 1970, 124 
pp, (ED 040 274) 



The research topics covered by this review of studies and 
literature published between 1962 and 1969 are arranged 
into four categories with a corresponding bibliography 
topically arranged, There are four focus areas: 

« The Stadent Learner s This section inalttdes a 
reviet^ of research concerned with self ^concept , 
work adjustment, and special needs-=data that 
may be considered as input for curriculum 
development and implementation in cooperative 
vocational education . 

• The Emplo yment Community i Manpower analysis 
studies, manpower data sources, task analysis 
models, and occupational analysis studies are 
reviewed in this section* 
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Educational Technology i Procedures , products , 
and problems of research dealing with instruc- 
tional objectives and content are discussed, 
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and a revlfe^^ of seLected research concerned 
with teaching techniques and :^xOCficlures is 
included* 

Program Itnplementation ; This section review; 
research and literature dealing with program 
administration and evaluation. 



Few of the studies reported dramatic evidence of the value 
of one instructional techniqua or one instructional syeten 
over another. Reasons identified were the lack of pre-- 
test/post^teet inBtrumentSiP techniquea anr:. procedures 
sensitive to measuring what should be TOaoured, and inade- 
quate deFign of research studies that corit:^rol for inter-- 
vaning variables* The author reports that the most sue-- 
ceasful studies investigated a single factor and provided 
obviously similar situations for the e^cperimental and 
control subjects* 

The author concludes that the basic concepts of cooperative 
vocational education need to be logically organized, arti^- 
culatad, and subjected to systematic tlicaory building, 
leading to the establishmerit of general guidelines. 
Research n#eds and priority areas identified by the author 
Includa the following^ 

• A theoretical and philosophical frafttework for 
research and development in vocational 
education 

• Interpretation dissemination oystema for 
manpov/er data to be used by program 
developers 

• Research technology to provide for the trans- 
formation of occupational analysis data into 
instructional objectives 

• Techniques and instruments for the evaluation 
of instruction 

• Study of the Gontrib^ition of occupational 
experiences to professional training and th© 
impact of teacher-coordinator characteristics 
on the behavior of the student-^learner 

0 Research involving psychological constructs 

such as "work values" and "personal adjustment" 

• Dlsseminatiori of research and dovelopmeiit products 
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Revlfev and Synthesis of Research on the EcQnomics of 
Vocational EducatJ^ori ^ Research Series No, 16, 
J. Robert Warmbrod, ~ Colurnbusi The Center for Vocational 
and Technical Education; The Ohio State University^ 
Novamb^r 1.968^ 62 pp. (ED 023 937) 

The doouttiMts included in this review ^ published between 
1962 and 1963, identify issues and problems in reaearch on 
the econoitiics of vocational education. The point is made 
that vhila there has been g-eneral agreeraejiit that education 

is a vital eleinent in eoonomiG grow^th^ the relative contribu 
tions of, ocouoational tsroarams and various other educational 
programs have not* been clearly shown, 

A discussion of the research investic^atlng education as it 
is related to earnings , rate of return on investment^ eco-- 
nomic grov^thf and e^cternal benefits is followed by a str^.ilar 
but more extensive treatment of research on the economic 
aspects of vocational education* The author points out 
that since there is concern not only for the efficieiit use 
of resources allocated to vooationai education but also 
for consideration of alternative prorrrams to reach the 
same objective^ costB of programs must be justified on the 
basis of otitcomes* Several approaches and techniques for 
measuring these outcomes are discussed; and specific atufl-- 
ies and critiques of studies using Qost-^benef it and cost^ 
effectiveness procedures in analysing vocational education^ 
technical education^ and manpower training prograins are 
described. Conceptual and practical problems involved in 
the application of cost-benefit analysis to vocational 
edueation are pointed out., and the suggestion is made that 
those who plan research in this area becorna familiar with 
the issues and problems discussed in the studies reviewed. 

The author contends that the usefulness of cost^benefit 
analysis as an evaluative technique in vocational eduoa- 
tion is liinitad by the requirement that benefits, as well 
as costs ^ be quantitatively assesmed in monetary terms. 
He suggests that cost^ef feativeneBs analysis is more appro- 
priate since it takes into account non-economic as well as 
economic bene fits ,^ The author concludes that research 
pertaining to the effectiveness of vocational education 
would be enhanced by greafceir joint efforts on the part of 
vocational educators and eaonomists * 
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Review ana Synthesis of Research on the Place ment and 

F Qllovr-up of Vocational E ducation studentsT J. Kenneth .s, 

LittleT' Columbus : The crenter for Vocational and .-j 
Technical Education, The Ohio State University, February 
1970, S4 pp. (ED 037 543) • j. 

The author character izes the studies reviewed as a cross ..j^ 

section of follow-up research reported between 1964 and | 

1969. Major fields covered in the studies include voca- .j 
tional education at the secondary, post-secondary, and 
adult levels. Special groups examined by the studies are 

Manpower Devslopment and Training program graduates, cor- ; 

rectional institution inmates, military trainees, the older . 
worker, and program early-leavers. One section of this 

report focuses on follow-up studies of vocationaL educa- ? 

tion graduates by program area— agriculture education, _ ,;; 
distributive education, health occupations, home economics,. .: 

office occupations, technical occupations, and trade, and . j 

industrial progranis. „" 

Among the findings reported are the following s -I 

• programs in vocational education are serving j. 
important segments of the population in areas 

not otherwise reached. 

• Those who move directly to jobs from vocational 
training have an advantage in earnings over 
untrainied cohorts . 

• Most vo>cationally trained persons like their 
jobs, especially if they find jobs for i^hrch 
they have been specifically trained. 

• The majority of vocationally trained high 
school graduates obtain employnert near the _ 
communities in which they recGivr. their training. 

• Graduates obtained their jobs primarily through 
their own efforts or those of friends and 
relatives. 

• In general, graduates from post-high school 
training programs have a clear employment advantafe • 
over high school program graduates. ,■: 

• While there is evidence that vocational progranis are 
probably worth their cost, this finding does not hold 
for all vocational programs, in all places, or m all 
fields. 
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The report concludes with these fiuggestions i 

• Follow-up and placement studies should be a 
component of the ©valuatior. of educational 
programB , 

• A systems approach should be used in evaluat- 
ing educational programs/ with purposes defined, 
outcomes described, criteria developed, and 
research designs adapted accorclingly* 

# Research is needed on employed persons who 
have acquired occupational skills and result- 
ing employment through non^school activities* 

# There is a need for more studies to which 
sophisticated statistical analysis can be 
applied. 



* * * 

* * * 



A Synthesis of Research Findings Which Describe Selected 
Benefits "and Outcomes for Participants in Vocational 
Education . Doug Sparkg, Washirigton , D * C , i Bureau of 
Occupational and Adult Education, Department of Health, 
Education and Welfare, October 197 7, 4 8 pp. 
(ED 147 576) 



The stated purpose of this study is to identify and explore 
selected benefits of vocational education. The findings 
of two categories of studies are eKplored^^major national 
studies and studies or reports from states or institutional 
sources* D^ta reported in these studies were gathered^ f or 
the most part, after 1970, 

Project Metro, the National Longitudinal Study of the High 
School Glass of 197 2, and Project Baieline are among tne 
national studies examined. State and institutional stud- 
ies and reports reviewed include an Arkansas follow-"Up 
study of 1970 graduates from eigtehigh schools, A Five - 
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Year FollQW--Up of Students Enrolled in Post- Secondacy 
Vocatlonal--Teahnical Transfer Programs by Noeth and Hanson^ 
an Ohio' school' district Btudy/ and a MassaQhusetts study 
of male high school graduates. Vocational educatiQn bene- 
fits examined include post-secondary sdhooling status and 
economio satief action values^ viewed, in terms of student 
satisfaction with education and employment and employer 
satisfaction with employees. 

The author points out difficulties encountered in inter- 
preting follow-up studies due to arbitrarily set intervals 
of time used for gathering data from program graauates and 
inconsistenGies in reporting categories from one state to 
another^ in definitions of terms , in sampling techniques, 
and in non-^response bias. 

With these shortcomings acknowledged^ the author offers 
these conclusionsi 

• Vocational graduates economically are doing as 
wall as or better than graduates of other 
curricula. 

• Vocational graduates are often mora satisfied 
with their jobs than academic and general 
graduates t 

m Vocational graduates consider their training 
an important contributing factor to job 
procurament , 

m vocational students who wish to continue 
education find openings available to them. 

m vocational education provides assistance to 
segments of society not adequately served by 
academic or general education* 

Two projects not discussed but noted by the author are 
The Development of Sample Designs tor the Follow-up of 
VocatiQnal Education Graduates ^ conducted by 
Weatat Research, inc. and aimed at proviaing sample designs 
to aid states in conducting follow-up studies, and A Study 
of the Comparison of the Benefits of Secondary and Post - 
Secondarv Vocational Education / conducted by the Montana 
State Department of Public Instruction to identify the psy^ 
chologlcal and sociological benefits of vocational education 
to students. "™ 

* * * 

* * * 
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What Happens Aftar Training i A Review of Follow-up of 
~ VQ cafe ional Gr adua t a a , Krishan Paul. Nashville s 
Urban Observatory of Metropolitan Nashville and 
university Centarsr September 30, 1916, 35 pp. 
(ED 140 010) 



The follow-up studies of vocational education graduates 
published between 1964 and 1975 and reviewed in this docu- 
mentr are arranged into four categories as follows i 

# Adminigtrative reportj^ t These are reports required 
of locals state ^ or federal administrators which 
may be baaed on nationwide or statewide surveys of 
former vocational students or on less formal inquiry 
by an instructor of a vocational program^ The 
reports include information about job placement^ 
wages / continuing educationi former trainee evalua-^ 
tion of educational and counseling services / and 
demographics ^ 

# Comparative studies g Research studies which evaluate 
individual vocational progfams by comparing them .. 
with other vocational pro.grams or with non-vocational 
programs are included in this category. 

d Cost-benefit analyses studies i Studies are reviewed 
that evaluate vocational education by calculating 
costs and benefits in an attempt to measure the 
increase in welfare or utility as a result of an 
educational program. 

# Procedural studies i This category includes studies 
related to^^ the" development of technology, question- 
naires and other instruments i data systems ^ 
methodology^ and guides and manuals that may be 
used in carrying out follow-up etudiea* 

The author points out the growing need that states have for 
follDw-up data to measure impact of vocational education 
on the labor market and the individual trainees. He feels 
that many of the present studies suffar from procedural 
and/or conceptual problems , such as inadequate data col- 
lection and sampling methods # a lack of adequate measures 
to assess the economic and non-*economic benefits of voca^ 
tional education, difficulty in separating the impact of 
non--sohool activities from educational or vocational acti- 
vities^ and the need for analysis of the relationship 
between vocational education and economic cycles^ 
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The report cecomrrtends sponsorship and encouragement of , 
research efforts at the regional, state, and national level 
as an essential part of planning and' evaluating the develop- ; 
ment of human resources necessary for industrial and economiG^ 
development. Two sample follow-up questionnaires are included 
at the end of the report—one for surveying foriner voca- _ 
tional students and the other for assessing employer satis- ;j 
faction with vocational graduate employeas. ■ •<^ 



* * * 

* * * 
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Categocy 3i EVALUATION METHO nOLOGY 
LIST OF TITLES 1 

Page 

Aeaountabil ity i Eva In atlon for Occupational _PrQ- 207 
' ^rams , Milo p. Johnson and Albert J. Rrafsky, 
Jr. American Technical Society, 1973, 

"Criteria Againit Which Voaational Education Should 208 
Be Held Accountable,'' Grant Venn, in Interpret- 
ing Outcome Measures in Vocational Education i 
A Final Report , Floyd l/ McKinney and others. 
The National Center for Research in Vocational 
Education, The Ohio State Universityi 1979. 

The Economic Evaluation of \^Qcational Training 209 
Programs V Manuel ^ynielnian. The Johns Hopkins 
Unxversity PreSB, 1976* 

Educational Evaluation and Decision ttaking . 209 
' Daniel L. Stuff lebeam and others, Itasca,. 

Illinois: P. E. Peacock Publishers,. Inc , ^ 1971* 

Evaluating occupation al Education and Training 210 
Prog ram s i Tim L. Wentling and Tom E, Lawson* 
Allyn and Bacon, inc,^ 1975* 

"Evaluating Vocational Education and Technical 211 
Programs," Jerome Mogs, Jr. and Ernst ^^* 
Stromsdorfer . chapter 9 in Vocational Education 
Today and Tomorrow , edited by Gerald 0. Somers 
ana Kanheth Little* center for Studies in 
Vocational and Technical Education, University 
of Wisconsin, 1971* 

Evaluating Vocational Education; Polices and 212 
Plans for the 1 97 0^^i th an Annotated Blbli-- 
ographF * Leonard A* Lecht. Praeger- Publishers , 
1974* 

Evaluation of Occupational Eiducation Pr ograrns, 213 
Jerome Moss, Jr, Oniversity of Minnesota^ 
Research Coordination Unit in Occupational 
Education, September 1968. 



Complete bibliographic citations Cor these titles, 
along with supplementary information, are provided with 
annotations on the pages that follow. 
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Evaluation of vocational Technical Edu cation. 
Phasa II . ^ew Educational DlrQCtlons^ Inc. 
Indianapolis i The Indiana State Board of 
Vocational Technical Education, May 1976, 

"Evaluation Rasearch in Vocational Education," 215 
J, Robert ^Tarmbrod. Beacon , American Voca-- 
tional Education Research Association, January 
1977. 

A Model for Evaluating Programs in Vocational 
Education for the Handicapped. Final Report . 
Gerald Bekker and James E. Christianeon, 
College of Education, Texas A^M University ^ 
August 1975. 

Practices in Program Evaluation i A Survey and So^e 215 
Case "Studies . Samuel Ball and ScarvTa BV 
Anderson/ Educational Testing Service, 
October 197 5. 

System for Statewide Evaluation of Vocational 217 
Education . Final Report ■ "Harold Starr and 
Ricliard Ai Dief f enderf er . The Center for 
Vocational and Technical Education, The Ohio 
State University, May 1972. 
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AccQuntabillty I Evaluation for Occupational Programs , 
" Milo p. "Johnaon and Albert J. Grafsky, Jr* Chicagos 

American Technical Society, 1973, IQl pp. (ED 087 902) 

This report addresses the development and utilization of 
accountability in occupational education and details a var-- 
iety of methods for measuring educational output and 
"reviBing strategies" for improvement. Accountability is 
defined as "the ability to demonstrate coat-effectiveness 

(efficient use of resources) in meeting predetermined edu= 
cational philosophy and goals, (1) when using verified 
objectives, (2) when serving well-^^identif ied students, and 

(3) when taking into consideration constraints relative to 
school, course, group of students, and community." , 

Fifteen goals of the occupational education accountability 
system are listed and guidelines are offered for preparing 
goals and objectivee for accountability based on a founda- 
tion of philosophy. Six measures of instructional output 
for occupational education are discussed and listed as 
follows: 

• Completion based on passing required minimum work 
as set forth in the teacher's course outline 

0 Completion based on measurable objectives which 
establish minimum standards of acceptable 
performance 

0 Units of credit plus grades—each based on 
measurable objectives and grading criteria 

• Completion with the student being 
adequately trained for employment 

m Placement in paid employment in occupation 
for which training was received 

• Follow-ups within first year of graduation^ 
including employer rating of the performance 
of the student graduate on the job 

* * * 

* * * 
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"Critaria Against Which Vocational Education Should Be Held 
AcGOuntable^ " Grant Venn, In Interpreting Outcome 
Measures in Vocational Education i A Final Report y 
Floyd l\ McKinney^ Kenney ET Gray, and Marie Abrani* The 
National Center for Research in Vocational Education, The 
Ohio State University, 1979, 250-205 pp. 



The author of this chapter suggeete that traditional criteria 
for accountability in vocational education are not relevant 
to the future and that the continued use of these criteria 
will result in decreasing the effectiveness of vocational 
education as a means of preparing individuals for future work 
just at a time when preparation for work is necessary for 
everyone and when many are facing changes in work roles. 
In selecting criteria for accountability, the following points 
are suggested for consideration i 

m criteria should be relevant to the individual's 
needs as well as to changing societal problemB, 

• A search for specific criteria that can be 
certified "to be right" may prevent the search 
for more appropriate criteria, 

# Criteria selected for accountability in response 
to student needs, iocietal problems^ and employer 
concerns are necessary before the evaluation 
process can be determined* 

Four broad categories of criteria were identified as 
follows: (1) instructional and program quality^ (2) Program 
relevance to individual and societal needs in relation to 
work; (3) Program impact on organization^ policy, support, 
and usage of vocational education- and (4) Individual transit 
tion to and growth in the work world* 

Venn suggests that if these broad areas for criteria of 
accountability seem appropriate, a national effort to define 
more specific criteria should be undertaken by a group com- 
posed of employers, parfentSi students, teachers # administra- 
tors , policy makers, and legislators, 

* * * 

* * * 
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The EcQnQmlc Evaluation of Vocational Training Programs . 
Manuel Eymelinan, World Bank Staff Occasional Papers 
No, 21, Baltimore^ The Johns Hopkins University Press, 
1976, 122 pp. 



The author etates that the purpose of this study is to pro-- 
vide a methodology for the assessment of vocational training 
programs that will enable policymakers to evaluate proposals 
for vocational schools or other forms of training and thereby 
to choose the most effective program for a particular situa-= 
tion. in particular, the study represents an effort to 
increase the efficiency of educational and training systems 
in developing countries. 

Advantages and disadvantages of on-the-job, off^the^job, and 
combined types of training programs are discussed in Chapter 2, 
A theoretical analysis of factors influencing the choice of 
a mode of training in Chapter 3 includes a discussion of the 
economic factors affecting the cost of training. Chapter 4 
furnishes data on programs in vocational schools in regard to 
hours spent in classrooms and in various laboratories, trana^ 
ferability of theoretical and practical instruction, and cost 
of laboratory equipment in 1972* Appendix B provides an 
analytic framework and a checklist of variables for review-- 
ing evaluations of vocational training, and Appendix C 
provides a review of studies evaluating vocational training 
in the United States and abroad* 



if ie is 
* * A 



Educational E valuatio n and Decision Making , Daniel L* 
"Stuff lebeam, Walter J, Foley, William J* Gephart, Egon 
Guba, Robert L. Hammond, Howard 0, Merrlman, and Malcolm 
M. Provus, Itaica, Illinois: F. E, Peacock Publishers, 
Inc. , 1971, 368 pp. 



Used as an education evaluation text, this book has four 
stated purposes I 
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• To expose five evaluation problem areas--def initionr 
decision-making r values and criteria, administrative 
levels, and the research model 

• To identify and assess eKisting formulations that 
may be useful in conceptualizing solutions to 
the five problems 

m To use the eKisting information to synthesiEe a 
new definition and methodology of evaluation 

« To provide guidelines for implementing the 
proposed new approach to evaluation 

The authors note that an effective and efficient evaluation 
requires a team efforti "The book is directed at those who 
commission the development of evaluation units ^ operate the 
units, use the information produced by the units ^ comprise 
evaluation teams, produce methods and materials for use in 
evaluation units, and train staff for work in evaluation units 



* * * 

* * * 



Evaluating Occupati onal Educati on and Training Programs . 
Tim l7 Wentling and Tom E* Lawson, Bostoni Allyn and 
Bacon, Inc., 1975, 355 pp. (ED 109 449) 



Designed to serve as a program evaluation handbook and guide, 
this book addresses teachers and administrators of occupa- 
tional education in public elementary, secondaryr and post-^ 
secondary programs as well as administrators and personnel 
connected with private instructional programs in schools and 
in industrial programs. The methodology described, however, 
is also adaptable to the evaluation of academic programs. 

The introductory chapter summarizes the history of evaluation 
and presents two current^ widely accepted definitions of 
evaluatioft — the decision-'oriented definition (Phi Delta Kappa 
committee on Evaluation) and the evaluator judgment definl-- 
tion (Worthen and Sanders) * 

Chapters 3 through 8 provide specific evaluation procedures 
for" student assessment, student follow-up, consultative team 
evaluation^ personnel evaluation, and cost analysis. 



210 

-^4 



Individual chapters provide specific evaluation activities and 
example forms and instruments to aid in the evaluation process 
The aonaluding chapter presents a general overview of how 
changes occur within education programs and how evaluation 
can be used to effect change and improve programs. 



* * * 



"Evaluating Vocational Education and Technical Programs^ " 
Jerome Moss, Jr. and Ernst W. Stromsdorf er . Chapter 9 in 
Vocational Education Today and Tomorrow . Edited by Gerald 
G. Somers and Kenneth Little. Madisom University of 
Wisconsin, Center for Studies in Vocational and TechnrQal 
Education, 1971, 221 pp. 



This chapter treats program evaluation as a process to 
determine the relative efficiency of a set of activities de- 
signed to attain desired outcomes. The process requires 
(1) that comparison between the outcomes of two or more sets 
of activities be made and (2) that differences found be 
attributable to variations in the program activities. The 
model, then, for evaluating the efficiency and effectiveness 
of vocational education should examine alternative processes 
which may produce the desired outputs; and it should determine 
which process is the more efficient. Evaluation charaGter-^ 
i^ed by this model is quantitative, directed at program 
purposes, and links costs with benefits , 

The problems associated with specification and measurement 
of input variables—student inputs, educational progess 
inputs, and socioeconomic influences — are discussed, as are 
the problems associated with specification and maasurement 
of program output and estimation of net effects of programs. 

The remainder of the chapter describes and reviews several 
kinds of non--economic studies of vocational education out- 
comes and economic studies of vocational education and 
manpower training programs. 



Evaluatin g Vocational Education; Policies and Plans for the 
1970s. With an Annotated Bibliography . Leonard A. Lecht. 
New York- Praeger^ Publishers, 1974 , 194 pp. (ED 123 395) 



This book, concerned with policy issues and related analytical 
problems involved in the evaluation of vocational education, 
is based on a study undertaken by the National Planning As- 
sociation for the U.S. Office of Education and the National 
Advisory Council of Vocational Education. The author points 
out that between 1965 and 1970 the federal share of the total 
expenditure for vocational education decreased from 29 per 
cent to 16 per cent and speculates on the future role of the 
federal government in determining priorities if support con- 
tinues to decrease. 

Reasons for data gaps and inconsistenelee in vocational edu- 
cation enrollment and student socioeconomic background infor- 
mation as reported in several national studies are discussed 
in Chapter 2. Subsequent chapters address the need for 
relating projected manpower needs to accurate data of enroj.1- 
ment in vocational programs in order to provide better 
planning information for both students and program .adminis- 
trators. Advantages and disadvantages of substituting 
revenue sharing for the grants system in distributing federal 
aid to vocational education are discussed in Chapter 5, 

The final chapter is based on Ernst W. stromdorf er ' s report, 
"The Methodology of Major Studies with Illustrations." _ 
Stromsdorfer points out that since many evaluative studies 
of vocational education do not employ accurate models for 
determining marginal costs, they fall to recognize distinctions 
between marginal and average costs and benefits. The model 
is quantitative and directly related to the specific purpose 
served by the program being evaluated. 

« It examines the nature of the output process 
of competing programs. 

• It determines which program and program output 
process is most efficient. 

The model requires the following steps: (1) Specification 
of program objectives; (2) Specification of processes or 
activities and the production function or process; (3) Speci- 
fication of cost function or cost relationship based upon 
the production function for each activity; (4) Specification 
of benefit functions based upon a set of indexes designed to 
measure program output; and (5) Comparisons of costs and 
benefits . 
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According to Stromsdorfar , the production function or process 
is a critical concept in any program evaluation since it is 
through understanding how program inputs affect outputs 
that rational changes in program structure can be made. The 
author of this chapter, Michael Carbine, contends that voca- 
tional education, particularly at the secondary level, should 
not be concerned merely with efficiency since the question of 
equity may thereby be ignored. Data drawn from the National 
Longitudinal Survey and studies by Teh-wei Hu, Fernbach and 
Somers, Eninger (Metro I and II), and others are used to 
discuss such issues as vocational education and earnings^ 
and employment, socialization, further education, remaining 
in school, and post-secondary and secondary programs. 



it ^ is 
* * * 



Evaluation of Occupational Education Programs . Jerome Moss, 
Jr. Minneapolis-. University of Minnesota, Research 
Coordination Unit in Occupational EduGation^ September 
1968, 22 pp.- = 



To provide a conceptual framework for evaluation of voca- 
tional education programs, the author discusses these eight 
dimensions of program evaluation! (1) The importance of 
program evaluation; (2) Some causes of past inactivity in 
evaluation; (3) A definition of program evaluation; (4) Pro-^ 
gram outcomes (or evaluative criteria); (5) Program charac- 
teristics; (6) Two roles of program evaluation; (7) Evalua-- 
tion as a part of the educational change process i and 
(8) Some research approaches to evaluation. Research ap- 
proaches discussed Include formative evaluations / follow- 
upS/ experiments^ interrupted time series^ and analysis of 
regression . 



* * * 

* * * 



213 

225 



Evaluation og Vocational Technical Education . Phase II. 
New Educational Directions, Inc. Indianapolis! The 
Indiana State Board of Vocational Technical Educati.on, 
May 1976, 17 pp. 

The vocational-technical education evaluation model described 
in this document is designed for use at the individual pro- 
gram level and consists of 13 main components as follows: 

(1) Descriptive Information 

(2) Demonstration of Need 

(3) Student Recruitment and Selection 

(4) Curtficulum 

(5) Guidance and Placement 

(6) Pacilltles 

(7) Program Staffing and Administration 

(8) Community Involvement 

(9) Youth Organizations 

(10) Cooperative Programs 

(11) Program's Effects on Students 

(12) Holding Power and Popularity 

(13) Satisfaction of Various Audiences with Program 

Each component, broken down Into subcomponents, consists of 
a general rationale for evaluating that aspect of the voca- 
tional program, suggestions for approaches to investigation, 
and recommendations for research and development. 



* * * 

* * * 
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"Evaluation Research in Vocational Education*" J. Robert 
Warmbrod. aeacon, Vol. 6, No, 1, January 1977 , 6 pp. 



In his addreis before the toierican Vocational Education Re-- 
search Association^ reprinted in Beacon , the Association's 
newsletter, Warmbrod proposes a comprehensive analysis and 
critique of recent state, regional, and national studies 
that "purport to investigate the effectiveness of vocational 
education in the secondary and postsecondary schools." 
Such a review should address the following questions i 

• What are the specific strengths and weaknesses 
of the design? 

• To what extent do the instruments and pro-- 
cedures used minimize measurement error? 

# Are the findings valid? 

# To what populations and programs can the 
valid findings be generalized? 

.Whether vocational education is viewed as a function of the 
total school program or as only a segment of the school pro- 
gram must be considered in designing and conducting evalua-- 
tion studies. Evaluation of a school's occupational educa- 
tion function is considered to be the description and eKpla- 
nation of a series of outcome variables in terms of a variety 
of input variables such as initial abilities and motives of 
students, family and community characteristics and values, 
school resources and policies, peer groups, the nature of 
the instructional program, and experiences of students in 
school and out of school* 

When the outcome variables relate to labor market entry and 
performance, factors over which the school has little con- 
trol (economic conditions, employment practices, mobility 
and migration of students) need to be considered. The ap- 
propriate research methodology is described as a "naturalis- 
tic' correlational method with regression analysis and other 
multivariate techniques the major statistical strategies for 
attempting to partial out the influence of the various in- 
puts on the criterion measures." 
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A Model for Evaluating Programs in VQaat i orial Education for 
the Handlcappedy Final Report . Gerald Bekker and 
James E, Cristiansen. College Station i College of 
Education, Texas A&M University, August 1975, 179 pp. 
(ED 115 984) 



The stated purpose of this study is to develop an evaluation 
model for use in determining the effectiveness of pilot pro- 
grams in vocational education for the handicapped and to field 
test the developed model by conducting evaluationB of 16 
pilot programs in schools and state hospitalB in Texas. A 
four-member evaluation team consisting of individuals knov/l-- 
edgeable in vocational education or in the field of special 
education for the mentally retarded coilected data by on- 
campus visits. 

The model was found to provide useful decision-making infor^ 
mation for comparisons on either a "prograin by program" or 
"institution by institution" basis i however^ goal concepts 
used in the model were found to be mora applicable to voca- 
tional education programs in public seaondary and post-seGondary 
institutions than to the pilot programs in state schools 
and hospitals. The model development process^ test proce- 
dures, and findings are detailed in the report; and sample 
questionnaires used in the study and related computer program 
information are included, 

* * * 

* * * 



practices in Prog ram Eva luations A Surv ey and Some Case 
Studies . Samuel BalT and Scarvla B. Anderson. Prince- 
torn Educational Testing Service, October 197 5, 152 p^. 
(ED 115 680) 



This report is .the second of a three-phase series to study 
and improve theory and practice of training/eduGation pro- 
gram evaluation. Phase two consists of two related research 
activities directed at collecting data about actual practices 
of program evaluation in various settings i the surveying of 
200 adult education/ training programs which had undergone sorie 
kind of program evaluation, and case studies o£ 14 progra-^a 
(including site visits) judged to have undergone the best 
evaluations , 

The 200 program evaluations were divided equally among four 
types of sponsoring agencies— Department of Defense? other 
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federal government departments and agencies; state and local 
governments and agencies / including junior/community colleges; 
and private business and industrial organizations. Approxi- 
mately 7 0 percent of the programs had been formally evaluated, 
usually for the purpose of program improvement rather than to 
assess program impact* The formal evaluation typically had 
consisted of queationnaires and interviews of students and 
teachers; and these evaluations usually had been called for , 
funded, and carried out by internal rather than external 
agencies. Written evaluation reports were rare, and written 
evaluation reports available to outsiders were even more rare. 

The first phase of this three-phase series resulted in the 
publication of the book, Encyclopedia of Educational 
Evaluation , which presents in non-»technical language the 
major concepts and practices in education evaluation* The 
third phase resulted in Prof essional Issue s in the Evaluation 
of Education/Training Programs t Ball and Anderson, October 
1975 (ED 115 681). Characterized as a "codification of 
evaluation principles" and a framework for appropriate eval^ 
uation practices, this publication provides evaluators and 
those who commission evaluations with checklists for a system- 
atic approach to evaluation. Checklists are included for 
common purposes of evaluation and appropriate methods of 
investigating each, sources of evidence, types of administra-- 
tion relationships, audiences for dissemination of results, 
value orientations of evaluators, competencies needed by 
evaluators, and ethical responsibilities of evaluators and 
related groups, 

* * * 

* * * 

A System for Statewide Evaluation of vocational Education . 
F ^al Report . Harold Starr and Richard A. Dief f enderf er . 
Columbus: The Center for Vocational and Technical Educa^ 
tion. The Ohio State University/ May 1972, 178 pp. 
(ED 066 561) 

This publication describes an evaluation systOTi designed to 
assist state divisions of vocational education in cooperation 
with local school systems, in assessing the effectiveness of 
vocational education programs. System components can be mod- 
ified by vocational agencies in the states to meet special 
needs in collecting data about local program characteristics, 
student status and characteristics, and student follow-up in 
terms of outcome measures* charts, forms, and tables illus- 
trate the system's potential for collecting data. Also in- 
cluded is an envelope of evaluation instruments designed for 
use in the state evaluation. 

* * * 

* * * 
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Category 4i EMglRICAL STUDIES FOCUSING 
ON VOCATIONAL EDUCATION 
TOPICS OTHER" THAN OUTCOMES 

LIST OP TITLES''' 

Eduamtlon for a Changing World of Worki Report of the 219 
Panel of Conaultante o n Voc ational Ed ucation. 
U.S. Department of Health r Education r and Wel-^ 
fare. Office of Education, 1963, reprinted 
1964, 

Eduaation for a Changin g World of Work, Appen - 220 
dix 1. Technical Training in the Unitad States , 
Lynn A. Emerson, U. S , Department of Haalth Edu- 
cation, and Welfare, Office of Education, 1963. 

Education for a Changing World of Work, Appen - 221 
dix 2p Manpower Farming and Related Occupations . 
C7 E* Bishop and George S, Tolley. U.S. Depart- 
ment of Health, Education, and Welfare, Office 
of Eduaation, 1963. 

Expectations and Satisfactions of parents and Stu - 221 
dents with Vocation al a nd Technical Education . 
Statewide Evaluation of Vocational^-Technical 
Education in Florida. Volume 4 . Gordon S* 
purrington/ Florida State University, 1972, 

Note s and Working Papers Concerning the Administra - 222 
~ tion of Programs Authorised Under Vocational 
Education Ac^df 1963 . Publxc Law 88-210, as 
Amended. U.S. Congress, Senate, Conmittee on 
Labor and Public Welfare, 1968. 

Training America's Labor Force i Potential, Prog - 223 
ress, and Probieins of Vocational Education . 
Report to the Congress . U\ S. General Accounting 
Office, "1972;" 

What Is the Role of Federal Assistance for Voca- 224 
^ional Educati on? Repo rt to the Congress . U.S. 
General Accounting Office, 1974. 



■'"Complete bibliographic citations for these titles, 
along with aupplementary information, are provided with 
annotations on the pages that follow. 
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Edueatlon for a Changing World of Work; Report of the Panel 
of Coniultants in Vocational Education (CE-80021) . U,S. 
Department of Healthy Education^ and Welfare, Office of 
Education. Washington, D*C.i U.S. Government Printing 
Office, 1963, Reprinted 1964, 196 pp. (ED 019 500) 



In 1961 a Panel of Consultants was appointed by the Secretary 
of Health, Education, and Welfare and charged with the re-- 
sponslbility of reviewing and evaluating the national voca- 
tional education acts in order to niake recoiranendations for 
improving and redirecting training for industry, agriculture, 
and other occupational areas. The Panel consulted varioue 
specialists, commissioned studies, and convened a number of 
special eonferences on the educational aspects of national 
manpower resources and requirements, the results of which 
are recorded in this publication. 

Federally reimbursed vocational training in the public schools 
in 1961 were found to consist of courses in agriculture, 
trade and industry, home econoinicB, distribution, praot^ 

ical nursing, fishery, and technical education. Support 
service areas included occupational information, vocational 
guidance, supervision and teacher training, research, youth 
organizations, instructional materials and administration. 
In studying the training programs and services in vocational 
education, the Panel found that use of federal funds for 
occiipational information and vocational guidance was extreme- 
ly limited. Research had been sporadic and uncoordinated 
and directed chiefly toward program operation* The federal 
level of administration in vocational education had changed 
in 45 years from direct responsibility to the President of 
the United States to the fourth level of administration in 
the Department of Health, Education, and Welfare. Additionally, 
a need was felt for instructional materials in vocational 
education programs, 

in reporting limitations, the Panel pointed out that voca-- 
tional education was not sufficiently responsive to labor ^ 
force demands. It was not available in many schools j train-- 
ing choice was limited; placement services to graduates were 
not adequate; and standardized reporting procedures were 
needed. The Panel also pointed out that the contributions 
of vocational education in international programs were small 
in comparison to probable need, and vocational educators were 
not generally involved in the initial planning for economic 
aid to other nations. 



219 

33X 



A-192 



The Panel's reconimendations were grouped under three "need 
for change" categories: 

• Vocational education must be made available to all 
people who have the need, the desire, and the 
ability to bentifit from such instruction. 

• Education for occupational competency should ba 
carefully correlated with the possibility of 
employment . 

• An evaluation comniittee of competent authorities 
should report the strengths, limitations, and 
weaknesses of the vocational training program. 

The Panel further recommended that since occupational 
categories are no longer isolated from each other funding 
and administration restrictions should not be permitted to 
prevent expansion of course content, combination of courses, 
or elimination of courses when such changes may provide a 
better training program. 

The Panel concluded that vocational education and general 
education are complementary and equally important to indi- 
vidual occupational competence and that expanded vocational 
education, apprenticeship, and technical training are 
especially needed to prepare both new workers and the unem- 
ployed to fill job openings for skilled or specialized workers 
Stimulus and support of vocational and technical education 
were viewed by the Panel as part of the federal government's 
responsibility in equalizing educational opportunities ^ and 
in providing the skilled manpower needed by the nation's 
economy . 



* * * 



Educat ion for a C hangin g World of Work . Appendix li 

Tech nical Training i n the Unite d States . Lynn A. Emerson. 
Washington, D.C.s U.S. Department of Health, Education, 
and Welfare, Office of Education, 1963, 170 pp. 
(ED 019 502) 



Appendix 1 to the report of the Panel of Consultants deals 
with the status of technical education in the United States 
in 1961. The technician is described as a worker giving 
technical support to and performing many of the tasks of the 
engineer. The Appendix lists various technical occupations 
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and tasks performed by technicians and deBcribes programs 
used for providing trained technicians to meet the needs of 
industry, specific recommendations are made related to the 
type of institution in which technical training is offered . 
General recoimnendationB for legislation in the field of tech 
nical education are included. 

* # * 



Education for a Changing World of Work. Appendix 2g 

Manpower Farining and Related Qccupat ions . C. E. Bishop 
and G7"s. Toliey, Washington/ D.C. T U.S. Department of 
Health, Education, and Welfare, Office of Education, 
1963, 51 pp. (ED 019 501) 

Appendix 2 to the report of the Panel of Consultants dis- 
cusses the effects of economic progress on the structure of 
the agricultural industry in 19 61/ the amount and quality of 
human resources employed in farming and related occupations, 
and educational implications of these changes. Decrease in 
number of farms and increase in agricultural technology, re^ 
suiting in lower demand for farm labor, led to projections 
of continued reduction in farm population into the 1980s. 
Trends with implication for agricultural education include 
the need for more highly trained farmers with managerial 
ability and agricultural curricula that reflect technolog-^ 
ical and occupational changes. 



* * * 

* * * 



Expectations and Satisfactions of Parents and Students with 
^^ Vocational-Technical Educa tion. Statewide Evaluation of 

Vocational-Technical Education in Florida. Volume 4 . 

Qordon s . Purrington . Tallahassee i Florida State 

university, 1972, 67 pp. (ED 074 225) 



This study, one of a series of studies conducted in the 
"Statewide Evaluation of Vocational-Technical Education in 
Florida," examines the expectations and satisfactions of 
students and their parents with secondary and post-^secondary 
levels of education in traditional public schools, vocational 
centers^ and in a coranunity college. 
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An analysis of data gathered from students o£ ten participating 
schools and the students' parents revealed that both students 
of vocational-technical centers and their parents supported 
these schools and wore more satisfied with the educational 
eKperiences offered there than were students of the tradi- 
tional high school and their parents. General support from 
all groups for vocational programs, particularly in the area 
of vocational guidance, led to the conclusion that greater 
emphasis should be placed on vocational guidance m the 
traditional high school. 



* * * 

* * * 



N otes and Working Papers Concerning the Mminist ration of 
Pro grams Authorized Under vocational Educatio n Act of 
1963, Public Law 88-210, as Miended . U.S. Congress, 
Senate, Comnittee on Labor and Public Welfare, Subcom- 
mittee on Education. Washington, D.C.s U . S . Government 
Printing Office, March 1968, 423 pp. (ED 021 151) 

Under the provisions of the Vocational Education Act of 1963, 
the Advisory Council on Vocational Education was directed 
to review the administration and status of vocational edu- 
cation programs and to make recomnendations for their im 
provement. The report Vocational Education s The Bridge 
B etween Man and His Work is divided into tnree parts (all 
r^nniiained in Notes and Working Papers Concerning the 
af^minlst ratlon of Programs ) . Part 1 is a summary of the 
general report contained in Part 2, and Part 3 includes 
recommendations adopted from the general report. The theme 
throughout the final section Is "Never before has^^attention 
to the individual as a person been so imperative." 

There were 26 specific recommendations for legislation, 
including the following: 

• Combine all federal vocational education acts 
administered by the Office of Education 

• Establish a Department of Edugation and Manpower 
Development at cabinet level 

• Provide funds for the Conunissloner of Education 
to make grants to or contracts with state boards 
and other public or non-profit private agencies 
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for planning # development , and operation of 
eKemplary and innovative programs for occupa^ 
tional preparation 

m Provide funds and require the Office of Education 
to be responsible for collactlng data and 
preparing an annual descriptive and analytical 
report on vQcational education to be submitted 
to the President and Congress 

m Provide that each state conduct a periodic 
statewide review and evaluation of its 
vocational education program 

• Include within the definition of vocational 
education "pre--vdcational" education and 
"employability skills" 

• Expand the definition of vocational education 
in the act to include the responsibility of 
education for initial job placement and 
follow-up 

• Establish two to four centers for currieulum 
development in vocational education. 



* * * 

* * * 



Training America 's Labor Force i Potential # Progress^ and 

Problems of Vocational Education, Report to the Congreae^ 
u/sr General Accounting Office, Washington, D\C. , 1972, 
47 pp. (ED 047 208) 

This document contains the findings , conclusions, and 
recommendations resulting from the investigation in 1972 by 
the CSeneral Accounting Office of high echool and community 
collage vocational education programs in California, Michigan 
Ohio, and PennBylvania , These four states, having received 
22 per cent of the vocational education funds alloted to 
the states in 19 72; were reviewed to determine whether or 
not legislative objectives were being met and to identify 
major problems* Topic headings under which information is 
reported include "Vocational Education Not Reaching All Who 
Need It," "Funds Targeted for the Disadvantaged Miss the 
Mark," and "Management Information Incomplete and Inaccurate, 
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fmong the reconunendations made to HEW are the following i 

• HEW should coordinate federal and state efforts 
to define the information needed to evaluate 
program results adequately. 

• Research should be undertaken regarding the axact 
nature and extent of the funding and image 
problems attached to vocational education in 
order to determine what action is required to 
achieve more fully the objectives of the 
Vocational Education Act, 

• Regional offices should closely monitor the 
use of federal funds for special programs 
and services for the disadvantaged to ensure 
that funds are being used properly. 



* * * 

* it 



W hat IB the Role of Federal Assistance for Vocational 

Educati on? Report to the Congress . U.S. General Account- 
ing OttxoB, Washington, D.C, 1974, 123 pp* (ED 105 132) 

TO review implementation of vocational education programs in 
1973 at national/ regional, state ^ and local levels, the^ 
General Accounting Office undertook this in-depth study in 
seven Btates---Callf ornia , Kentucky, Minnesota, Ohio, 
Pennsylvania, Texas, and Washington, The study focused on 
these questions i (1) What role does the federal dollar 
play? (2) How is vocational education planned? (3) How 
are federal voGational funds dietributed? (4) How are 
training resources used? (5) Is training related to 
employment? 

Among the recomnendations submitted to the Secretary of the 
Department of Health, Education, and Welfare are the following 

• HEW should develop with the states an improved 
approach to planning which will better meet 
state needs and provide necessary information 
to monitor and evaluate federal prograni 
eKpenditures • 
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• HEW should expand its effort to enforce the require- 
ment that all local and state education agencies i 

in planning vocational programs ^ identify the 
needs of studente and those of public and 
private business ^ industry, and labor ; 
these needs should be considered the primary 
basis for decision making in the provision 
of vocational services supported by the 
Vocational Education Act. 

• HEW should increase its efforts in the development 
of vocational information systems that will 
provide comparable data and should continuously 
review utilization of that data to improve 
vocational education . 

• HEW should analyse actual state practices in 
distribution of federal funds to determine 
consistency with the law's criteria. 

• HEW should assist states in identifying and 
implemanting strategies in order to allminate 
barriers which inhibit improvement or expan^ 
sion of vocational programs and which restrict 
persons from fully participating. 

• HEW should implement provisions of Title IX 
of the Education Amendments of 1972 to 
eliminate sax discrimination in vocational 
education^ particularly by encouraging the 
use of techniques that have proved effactive 
in recruiting members of one sax to occupa- 
tions traditionally considered the prerogative 
of the other sax. 

The following recommendations were made to the Congress for 
consideration in deliberations on the Vocational Education 
Act; 

• Require federal funds to be directed to programs 
for which existing or anticipated job oppor-^ 
tunities can be demonstrated 

m Require work experience to be an integral 
part of the basic grant programs, to the 
extant that it is feasible 

• Require schools to take responsibility for 
job^placement assistance and follow-up 
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It was further recommendecl that congress (1) amend provisions 
of the Higher Education Act of 1965 so that students without 
a high school diploma or equivalent would be able to take 
advantage of federal grant and loan programs for post- 
secondary schools and (2) consider amending the Vocational 
Education Act to remove restrictions limiting vocational 
education opportunities to those in or above ninth grade 
since not all handicapped youth are able to reach the 
secondary level . 



* * * 

* * * 
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Category 5; MISCELLANEOUS 
LIST OF TITLES^ 

Caree r Intern Program i Final Report. Volume 1; An 229 
" — Experiment in Career Education that Worked . Michael 
Langsdorf and Richard A. Gibboney. U.S. Department 
of Health, Education, and Welfare, National Insti- 
tute of Education, May 1977. 

Education Amendments of 1976. Public Law 94-482 . 229 
Ninety-fourth Congress, October 12, 1976. 

Education Amendments of 1976 and Their Implications for 231 
Vocational Education . William W. Stevenson. The 
Center for l^ocational Education, The Ohio State 
University, 1977. 

Ed ucation and Job Satisfaction g A Questionnable Pay - 232 
— oii. Robert P. Quinn and Martha S. Baldi de 

Mandilovitch. U.S. Department of Health, Education, 
and welfare. The National institute of Education, 
March 1977. 

Equality of Educational Opportunity . James S. Colenian 23 3 
and others. National Center for Educational Statis- 
tics, 1966. 

Equality of Educational Opportunity, Reconsidered . 234 
James S. Coleman. Johns Hopkins University, 1967. 

Evaluation of the cumulative Effects of Research for 234 
Better Schools Career Education, 1974-19'7l1 Thomas 
W. Biester. Career Education Program, August 31, 1976. 

Factors Related to the Coordination and Effectiveness 235 
o f dccupational Education Programs. Final Report . 
John T. Pelham. Social Science Research Center, 
Mississippi State University, August 10, 1972. 

Hearings Before the Subcommittee on Elementary, Second - 236 
a rv, and Vocational Education of the Committee on 
Education and Labor. Volume 1 . House of Representa- 
tives, Ninety-fourth congress. First Session on H.R, 
19 and Related Bills to Amend the Vocational Bduca- 
tion Act of 1963, February and March 1975. 

Hearings BefOi.a the Subcommittee on Elementary » 237 

Secondary ,_ and Vocational Education of the comniittee 

on Education and Labor. Volume 2 . March, April, May, 
and June 1975. 



Complete bibliographic citations for these titles, along 
with supplementary information, are provided with annotations 
on the pages that follow. 
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How Effective Is Schoollnqf? A Critical Review and 5yh - % 3 7 
the s 1 B o f Re sear ch F indlnga . Harvey A. Averch and 
others. Rand Corporation, March 197 2. 

...» 

Idsntifying Handicapped Students and Their Vooatlbhai 238 
Nafeds for 1977-1982" . Marion E. Pranken* Madison s 
Wisconsin Vocational Studias Center^ University of 
Wisconsin^ July 1977, 

Manpower EoonQinic Education and the Transition from 239 
School to "Work , Robert L. Darcy arid pthers* The Ceh- 
tar for Economic Education^ Colorado state Univarsity^ 
Pabruary 1974, 

Ph ilosophy for Quality Vocational Education Programs . 239 
Fourth Yearbook of tha American Vdcational Aesocia-^ 
tion. Edited by Melvin L. BarloWi The Amaricin Voca^ 
tional Associationf Inc., 1974, 

RBS career Education, 1975-1976 i Wlthin-Model Evalua - 240 
" tion Raport l Thomas Biester. Research for Better 
Schoolsr Inc* f August 31, 1976. 

Rural oriented Research and Davelopment Projects: A 241 
" Raviaw and Synthesis , RaD Monograph 50- U.S. Depart- 
ment of Labor, Employment and Training Administra- 
tion, 1977* 

Sex Fairness in Career Education , Maria Peterson and 241 
Louise Vetter. The Canter for Vocational Education , 
The Ohio State University, 1977. 

# 

Sex Fairness In Vocational Education . John R. SchenGk. 242 
The Center for Vocational Education, The Ohio State 
University, 1977. 

A Study of Our Nation's School s. George W. Mayeske and 24 3 
others: U.S. Departnieht of Health, Education, and 
Welfare, Office of Education, 1972. 

Vocational Education for the Handicapped i A Bibllog- 243 
rap hy of ERIC Documents . Paul E. Schroeder. The Cen- 
ter for Vocational and Technical Education, The Ohio 
State University, 1973. 

Vocational Education for the Handicapped; A Review . 244 
Marc E. Hull. The Center for Vocational Education, 
The Ohio State University, 1977. 

Youthi Transition to Adulthood (Report of the Panel on 244 
Youth of the President's Science Advisory Conmiittee) . 
James S. Coleman and others. Chicago i University o£ 
Chicago Press, 1974. 
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Career intern Pro gram; Fina l Report, Volmne Is An 
EKperimant In Career EduaatrQn That Worked . Michael 
Langidorf and Richard A* Glbboney, NIE Papers in Educa« 
tion and Worki No* 7. Washington^ D.C.i U,S, Department 
o£ Health, Education^ and Welfare, National Institute of 
Edueation, May 1977, 217 pp. (ED 142 795) 

The Career Intern Program (CIP) was initiated by the 
Opportunities industrialization Centers of America, Inc, 
with support for development and evaluation from the U.S. 
Office of Education and the National Institute of Education. 
The Career intern Program is described as aombining basic 
and career education for high school youth who are not suc- 
caeaing in regular school. In addition to discussing the 
education problems of the program, the report deals with the 
effort to link formative and^ sununary evaluation to program 
development and utilization. 

Tachnieal aspects of the evaluation design with a description 
of the instruments used and a discussion of the data analysis 
procedures are contained in a separate volume, cited below. 

career Intern Programi Final Report. Volume II. Technical 
Appendices . Richard A. Gibboney Associates, Inc. NIE 
Papers in Education and Worki No. 7. Washington, D.C.: 
U.S. Department of Health, Education, and Welfare, National 
. Institute of Education, May 1977, 231 pp, (ED 145 231) 
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Educ ation Amendments of 1976, Public Law 94-482. U.S. 

congress. Washington, D.C i U.S. Government Printing 
Office, October 12, 1976. 

The stated purpose of Title II (Vocational Education) of 
the Education Amendments of 1976 is as follows: (1) to 
extend^ improve, and, where necessary, maintain existing 
programs of vocational education? (2) to develop new pro-- 
grams of vocational education i (3) to develop and carry out 
programs of vocational education within each state that will 
overcome mm discrimination and sex stereotyping in voca- 
tional education programs; and (4) to provide part-time 
employment for youths who need the earnings in order to 
continue their vocational training on a full'-tim© basis. 



Among the salient features of the Amendments are the 
following! 

• Each itate must assign full-time personnel to 
assist in reducing sex discrimination and sex 
stereotyping in vocational education programs 
and activities. Each state is to expend 

"150,000 from the basic grant for this purpose. 

• Each state shall increase its advisory council 
from 12 to at least 20 members appropriately 
representing special groups. 

• Each state must maintain on file with the 
Commissioner of Education a general applica- 
tion containing twelve assurances covering 

a broad range of administrative and fiscal 
matters. The application includes the 
assurance' that the state will give priority 
in distributing funds to economically de- 
pressed areas and areas with high unemploy- 
ment, to programs which are new to the area, 
and to local education agencies with a high 
concentration of low-income populations. 

• At least 10 per cent of the state's allotment 
is to be used to pay up to 50 per cent of the 
costs of special programs, serviceBf and activ 
ities for the handicapped,' and at least 20 
per cent of the state's allotment is to be 
used to pay up to 50 per cent of the costs of 
special programs, services, and activities 
for the disadvantaged, for persons with 
limited English-speaking ability, and for 
stipends for students with acute economic 
needs which cannot be met under other programs 

• In program evaluation, employer feedback on 
the training preparedness of individuals 
shall be considered as well as placement data. 

• The state must use 20 per cent of its 
allotment for program improvement and suppor- 
tive services such as research, innovative 
programs, and curriculum development programs. 



m The state must use funds for programs in 
consumer and homemaking education. The 
federal share is 50 per cent except in 
econoinically depressed areas where it shall 
be 90 per cent* 

• There must be a greater coordinated effort 
involving seaondary ^ post-^secondary , adult 
education^ area vocational centers, and 
CETA programs. 



* * * 
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Eduoatlon Amendments of 197 6 and Thei r Imj^llcations for 
Vocational Education . "Information Series No. 122, 
William Stevenson. Columbus i The Center for Voca-- 
tional Education^ The Ohio State University^ 1977, 21 
pp, (ED 149 191) 

This information analysis paper, examining the Education 
Amendments of 1976, focuses on the areas of special legis'* 
lative emphasis that are expected to bring about the greatest 
change in vocational education. Topics receiving special 
attention include coordination with CETA; compliance with 
Public Law 94-142, the Education for All Handicapped 
Children Act; planning, information systems, and evaluations; 
elimination o£ sex bias and sex stereotyping; and special 
groups^^disadvantaged, handicapped, and English deficient. 

The author suggests that the legislation is an Indication 
that "Congress is no longer satisfied with a vocational 
education program that accurately reflects the labor market 
with all its exclusions, restrictions, and traditions" and 
speculates on the potential for changing the work place by 
changing the training programs which prepare students for work 
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Eduaatidn mnd Job Satlg factlon^ A Que stionable Payoff > 
" Rabart P. Quinn and Martha S» Baldi de Mahdilovitch » HiE 
Papers in iducation and Worki No* 5, Washington # btd.i 
U*S, Departmant of Healthy Education^ and Welfare i^he 
National Institute of Education^ Mareh 1977^ 69 pp. 
(ED 129 702) 

This report aKaminas (1) the assumption that the better the 
aduoation, the greater the chances of sacuring a desirable # 
satisfying job and (2) the social and paycholbgidal procaeies 
that may link education and job satisfaction. Sourcies of 
information used for the study include siKtefen previously^ 
published research reports bearing either directly or indir- 
ectly on the relationship between education and job satig-* 
faction and secondary analyses of nine national survteys 
designed for purpoaee other than underBtanding the relation- 
ship between education and job satisfaction. 

Five of the sixteen studies found a positive associatian 
between education and job satisf actionr three found a neg*^ 
ative association r and eight reported the relationship to bfe 
either non-eKistent or equivocal* Analyses of the nine 
national surveys found no increment in job satisfaction ^ with 
sucoaeding years of education. However, while no relations- 
ship was found between education level and job satisfaction 
among workers who had not gone to college^ those who had 
obtained college degrees were consistently mote satisfied 
with their jobs than were employees without degrees i All 
but one of the surveys identified "dredentials effect^" evi* 
denee that there was no payoff in job satisfaction from having 
college training unless one also received a college degree. 
Level of education was significantly and positively related 
to overall quality of employment i and the greatest increment 
in quality of employment occurred at those points where 
educational credentials were conferred* 

The report concludes with several recommendations for further 
research and for policy changes on the part of employers 
and educators. Among the reconunendations for policy changes 
are the fol lowing s 

• Employers and educators should be aware of 
the occupational needs of the ovar-^aducated * 

• The educational requirements establiehed for 
jobs should be re--eKamined, 

• Job design, where possible r should take into 
account the increasing education level of the 
labor force , 



m Iducators should place greater emphasis on general 
skills, anticipating the many job changes in life. 

• Training for specific jobs should be reserved 
until it becomes neceBsary for the worker to 
receive such training - 

m Educators should not justify "every unpleasant 
thing in school" as essential for securing a 
good job. 



* * * 
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Equality of Educational Opportunity ^ James S. Coleman # 
firnast Campbell/ Carol J. Hobson, James MoPartland, 
AleKander M. Mood / Frederio D. Welnfald^ and Robert L* 
York* Washington, National Center for Educational 

Statistics, 1966. 746 pp. (ED 012 275) 

This volume, the product of an eKtensive survey request^ 
in the Civil Rights Act of 1964 , reports on educational op-^ 
portunities in public schoolB for minority groups — ^Blacks 
American Indians, Oriental Miericans, Puerto Rieans, and 
Bdexican Americans, The survey addresses four major 
questions; 

(1) To what extent are the racial and ethnic 
groups segregated from one another in the 
public schools? 

(2) DO schools offer equal educational opportunities 
according to criteria regarded as good indicators 
of educational quality? 

(3) How much has the student learned as measured 
by performance on standardised tests? 

(4) What is the relationship, if any, between 
students' achievement and the kinds of 
schools attended? 

James Coleman of Johns Hopkins Univeriity held the major 
responsibility for the design t admlnlitration, and analysis 
of the survey, which was carried out by the National Center 
for Educational Statistics. 
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Initial findings from analysis of the data Indicated th%t 
Black students and teachers were frequently segregated _ from 
their white counterparts. Additionally, with integratipn 
the average minority student's test scores improved. However, 
the authors, acknowledging that the various groups of students 
might have been from different backgrounds, noted that this 
possibility had not been taken into account. 



* ★ * 

* * * 



Equality of Educational opportunity. Reconsidered ' james 
S. Coleman. Baltimore i Johns Hopkins University » 1967, 
19 pp. (ED 015 893) 

In eKamining the concept of equal educational opportunity, 
Coleman points out the need to consider intensity of the 
effects of school resources as well as the equality of the 
distribution of these resources. Focusing attention on 
equality of distribution of education resources, he con- 
tends, results in inattention to an important area of 
inequality, that of opportunity— not from the school system 
but from outside and "not overcome by the school system. 

He concludes that if a child's opportunity is to be separate 
from accident of birth into a given family, society must 
move toward providing an equality of opportunity based on 
the distribution of educational resources and the intensity 
of effect of those resources. 



* * * 
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Evaluation of the Cumulative Effe cts of Research for Better 
S chools Career Education , 1974-1916. Thomas W. Blester . 
Philadelphia, Penn.i career Education Program, RBS, 
August 31. 1976. (Funded by P. L. 8910, Title IV and by 
NIE Contract #NE-C-004-0011 . ) (ED 138 822) 

This report examines the effects of participation in the 
Research for Better Schools (RBS) Career Education Pro|ram. 
The sample for the study was drawn from 10th, 11th, and 12th 
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grade students enrolled in 1974 in a large Philadelphia high 
school containing the Academy for Career Education. Students 
were randomly aBSigned to experimental and control groups. 
The experimental group participated in the Career Education 
Program while the control group continued in a regular high 
school curriculum, instruments used to collect information 
on career skills, life or self skills, basic academic skills, 
and demographic characteristics included specially designed 
questionnaires, as well as previously developed inventories 
and scales and the standardized "Comprehensive Tests of 
Basic Skills*" 

compared with the control group, the eKperimental group 
demonstrated significant growth in career knowledge over 
the course of two years and showed significant growth on 
all basic skills measured. 



* * * 
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Factors Related to the Coordination and Effectiveness of 
Occupational Education Prograrns. F i nal Report . John T. 
Pelham. Jackson 2 MissiBsippi State University, Social 
Science Research Center, August 10, 1972, 196 pp. 
(ED 068 649) 

The primary purpose of this study was to develop and test a 
model for the study of job creation, job training, and job 
placement in the aommunity. Personal interviews with 8 4 
community leaders, agency representatives, and major employ- 
ere in four communities were conducted to gather manpower 
development process information. Additional data were gath- 
ered by participant obBervation, content analyses of documents 
and from the census and training agency records* 

Recommendationi resulting from the study include the following 

• To avoid duplication of effort, job training 
agencies must have mutual awareness of service 
offerings . 

• Members of the "power structure" must be 
aware of job training programs in the local 
community. 
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• Representation on advisory councils by coimnunity 
leaders and major employers should be encouraged. 

• Formal evaluation of vocational^technical 
training programs should include information 
from local employars- 

• Advisory and craft committees should be better 
utilized , 

• Training programs should be designed for 
open entry/open exit* 

• Administrators of vocational education should 

be encouraged to finance more inter-disciplinary 
research, symposia, and new learning experiences. 



* * * 

* * * 



Hearings Before the Subcommittee on Elementary, Secondary y 
and Vocational Education of the Committee on Education 
and Labor. "Volume 1 ,~ 94th Congress ^ House, First 
Session on hTjr/ 19 and Related Bills to Amend the Voca- 
tional Education Act of 1963. Washington, D.C.s U*S, 
Government Printing Office, February and March 1975, 

This volume Includes prepared . statementB , doauments, and 
testimony from representatives of various states, special 
interest groups, and associations before the House sub- 
committee ohaired by Representative Carl D. Perkins, The 
General Accounting Office (GAO) report^ What Is the Role 
of Federal Assistance for Vocational Education ? is reprinted 
in this volume, along with responses from the U,S. Department 
of Health, Education, and Welfare and the seven states 
covered in the GAO study. 
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Hearings Before the Su bcomyn lttee on Blementaryr S ecQnd^^ 
and VQCational Educa tiQn of the C oMiiibtae on Edu catj-on 
and Labor, Volume 2 , '94th Congress ^ House/ First 
Session on H.R, 19 and Related Bills to Miend the 
Vocational Education Act of 1963. Washington/ P.C. i 
U.S. Government Printing Office^ Marchi April, May^ and 
June 1975. 

This second volume of prepared statements, documents, and 
testimony before the House subcommittee of the hearings to 
amend the Vocational Education Act of 196 3 contains among 
its reprints documenta subinitted by Roman Pucinski, Rupert 
N. Evans, William G. Conroyi Edwin Herr, Norman C, 
Gysbers, and Terrel H, Bell, 



* * * 
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HOW Effective Is Scho Qling? A critical Re view and_ Synthesig 
of "Research Findings ." Harvey A. Averch/ Stephen J. 
Carroll"/ Theodore S. Donaldson, Herbert J, Klesling/ and 
John Pincus. Santa Monica, California i Rand Corporation^ 
March 1972, 222 pp. 

This report was prepared Cor the President's Commission on 
School Finance as part of its study of alternative funding 
arrangements for primary and secondary education. The 
objective was to asseis the current state of knowledge re-^ 
garding the determinaiits of educational effectiveness* 
Five basic approaches used in educational research were 
identified — •input/output , process^ organizational ^ evalua- 
tion, and experlentlai , The authors concluded from their 
review of research studies that ''research has found nothing 
that consistently and unambiguously makus a differencx^ in 
student outcomes*" Six major Issues toward which futura 
research could profitably be directed were identified. A 
15-page bibliography is included. 
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Identifying Handicapped Studen ts and Their Vocational Needs 
for 197 7-1982 . Marlon E. Pranken. Madisons VfxaconeLil 
Vocational Studiee Center^ University of Wisconsinr July 
1977, 65 pp. 

This report suimnarizes research conducted in Wisconsin 
during the fiscal year 19 77. Included in the report is 
information concerning research objectives, methodology, 
instrumentation, and sanipling design. The study, conducted 
to identify the cognitive, peychomotor/ and affective char- 
aetariatics of handicapped students 14^21 years of age in 
Wisconsin, surveyed a 20 per cent sample of the teachers of 
thaSB students* 

Data gathered indicated that the majority of the students 
were Engllsh"-Bpeaking Caucasian young meni The average 
intelligence quotient we below 80, and the students were 
generally belo\^ level in grade placement and achievetnent. 
Their career goals vimtm in the areas of agriculture^ manu-^ 
facturing, consumer and homemaking education, and con-- 
gtruction, The report included the following recommendations 

m DeciBion^maKing administratorB and planners 
should prepare educators for the kinds of 
students who w^ill enter vocational prograins 
at the secondary or post'-secondary level. 

• Strategies need to be planned to assist the 
handicapped in the coiMiunlty and in the school • 

• pre-service and in-service course experiences 
should be provided for vocational educators to 
learn special education techniques, and special 
education personnel should be provided to assist 
vocational educators . 

• A determination of how. these students have 
been affected should be made by recording 
their reason for leaving a program of 
instruction* 



* * * 
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Manpcwer Econonilc Education and the Transition fro m School 
to Work . Robert" L . Daroy , Richard V * Kauf finan / and ^ 
Edward P . Milker. Fort Collins n Colorado State Uni- 
versity, Dapartmertt of Economics, Center for Econoinic 
Education, February 1974, 217 pp. (ED 091 581) 

U^ing longitudinal data gathered over a five-'year period, a 
sample of 242 was drawn from 6 45 young female and male stu^ 
dents for study to determine the impact of an experimental 
manpowar economics course on world-^of-^work understanding, 
attitudes, and education^-related and employmsnt-related hm- 
havicr. While the eighth-grade course Jail semester 
1967-68) had significant short-run impact on understandings 
and attitudes of students who received instruation compared 
with those of control students matched for itiental ability, 
few long--run differenGes were observed between the groups 
during high school and as of the February 197 3 survey week 
eight months following graduation, 

Wage rates and v/eekly pay were not associated with scores on 
a test of world-of-wprk understanding or with IQ, academic^ 
rank in high school, or family income but were associated 
with labor union membership and sex* Both the experimental 
arid control groups made a smooth transition frorn school to 
work or to post-=secondary educational programs; and labor 
force participation, wages, and work satisfactian were rela- 
tively high and unemployment rates low* On the other hand, 
major differences were observed between graduates and dropouts. 



* * *■ 
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Philosophy for Quality Vocational Education Progcame. 

Fourth Yearbook of the "American vdaational Association* 
Melvin L* Barlow, ed. Washington, D.C* % The Mnerican 
\rocational Association, Inc., " 1974, 283 pp, (ID 102 425) 

This &nerican Vocational Association Yearbook deals with the 
relationship betwerm the philosophical concepts and pract-^ 
icel demands of program organization and operation in voca- 
tional aducationp Experienced educators present their 
differing philosophic orientations on a variety of topics. 

Melvin L, Barlov? views the role of vocational eduaation in 
an industrial society from a historical perspective and 
presents the yearboolc theme. Robert Miller examines the 
place and function oC vocation^il education in the total 



educational aystein. Garth L. Mangum presents the economics 
of vocational eaucation in a social context for oost/benefit 
analysis, anfl Gordon I. Swanson focuses on the develtppmant 
of vocational eaucation policy. instructi anal principlei, 
statUB of career guidance in America, learner characteristics, 
job placeinent and follow-up, and the place of community 
involveinent In education are other topics addressed. 

In the epilogue Barlow reviews relevant iisues and stiKesses 
the need for a recoimtiitnient by all vocatLonal educators to 
the basic priuciples and philosophy of vocational education. 
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RBS career Educatio n, 1975-197 6; V?ithln-Mo del Evaluation 
Repoftr^ThoniaB W. Biester. Philadelphia; Research 
for Better Schools, Inc., August 31, 1976, 50 pp. 
(ED 138 820) 

This report dsscribes the implernentation of Research for 
Better Schools (RBS) Career Education at three sites 
during the 197 5-7 6 school year. The program for secondary- 
level students uses three instructional conponents — Career 
Developinent, Career Guidance, and Basic Skills— to achieve 
program goals of providing students with cognitive skills, 
career ejcperlencss , and personal perspective to aid them m 
selection and pursuit of adult life goals. 

Modeled after the RBS Career Education prograni in Philadelphia' 
Academy for Career Education in operation for fovr years, _ 
programs were initiated in Washington, Pennsylvania? Washington 
Township, New Jersey; and Wilmington, Delaware. Common evalu- 
ation activities conducted at all RBS Career Education sites 
during the year included monitoring program operations and 
assessing participant opinions. In general, responses indi- 
cated that students, parents, and community participants 
were in factor of the program and recommended its continua- 
tion. 



* * * 
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Rura l Oriented Research and Development Proiects; h Review 
~ ^nd SynLfchesia. R&D Monograph No, 50. U,S. Department o£ 
Labor # Employment and rraining Administration* 
Washington, D,C.: U.S. Department of Labori 1911 , 134 pp. 

Based on a review of 71 documents, this report describes the 
results of research on ernployment and training problains and 
on programs in rural areas between 1963 and 1975. Major 
topics covered by this review, conducted by the Office of 
Research and Development of the Employment and Training 
Administration, Include the various definitionB of "rural" 
implied in the literature, factori affecting and teahniques 
uped in forecasting supply and demand for labor in rural 
areas ^ and findings and reiulta of research and development 
projects in rural areas. 
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S ex Fairiiess in Career Education. Information Series No, 
109* Maria Peterson and Louise Vetter, Columbus: The^ 
Centex for Vocational Education, The Ohio State University, 
1977, 43 pp. (ED 149 179) 

The first section of this paper addresses the question, "Why 
should we be cQncerned with sex fairness in career educa- 
tion? Subsequent sections focus on the current status of 
women ia employment and leadership positions in career 
eduQatlon* 

Suggeationa for making career education "seK fair" include 
the following i 

• Programs federally contracted should be 
conducted to teach women how to become aware 
of contract and grant announcements, how to 
prepare proposals, and how to manage contracts 
and grants, 

• Criteria for selecting state leadership personnel 
should be widely disseminated. 

• Leaders in career education should take the ^ 
initiative to nominate and promote outstanding 
^omen to serve on the Advisory Council. 
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Sex Falrnees In VQcational Education i information Series 
~ 120. John R. SGhenck. Columbus i The Center for 

Vocational Education, The Ohio State University, 1977, 
54 pp. (ED 149 189) 

This report, a review and synthesis of significant literature 
on seK bias and seK'-role stereotyping in vocational education, 
discuSBBS the impact that atereotyping and bias have on 
careers in vocational education^ the courses that have 
promoted sex-role stereotyping^ — socialization, instruGtional 
materials^ vocational interest inventories, and the world 
of employment; and the forces that promote eradication of . 
sex unfairness in vocational education* 

To reduoe sex unfairness in vocational education, the author 
recommends the following policies i 

• Enforcement of appropriate laws 

• Administrative support from the vocational 
education hierarchy in each state 

• Affirmative action in promoting vocational 
programs 

• Provision of new models to encourage young 
people to widen their OGCupational consider- 
ations and aspirations 

• Requirement of in- service education for 
vocational educators 

a A re-examination of the principles of 
vocational education 

m Dissemination of information to help 
eliminate sex unfairness 



* * * 
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A Study of Our Nation's Schools . George W. Mayeske, Carl E. 
Wieleri Albert E. Beaton, Jr w Prederlc D. Weinfeld^ 
Wallace Cohens Tetsuo Okada, John M. Proshek^ and 
Kenneth A. Tabler. Washington, D.C.i U.S. Department 
of Health, Education, and Welfare, Office of Education, 
1972. 126 pp. (ED 082 312) 

Intended as a eequel to the Coleman report. Equality of 
Educational Op portunity, this study was deBigned to discover 
which character'iBtiGS of the nation's schools are most close- 
ly related to school outcomes. A five per cent stratified 
cluster sample v^as taken which resulted in the inclusion in 
the study of about 650,000 students from approximately 4,000 
schools* Information was also collected from their teachers # 
principalB, and superintendents. The questionnaire items _ 
related to student social background, school characteristics 
(facilities, programs, policies, personnel, and personnel 
ejcpendituras) , and school outcomes (student attitudes, 
motivations, and achievement) • 

The study conaluded that, although schools are important, their 
influence is bound up with student social background. To 
gain more knowledge off specific sources of influence, experi-- 
mental studies of educational innovations were recommended. 
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Vocational Education for the Handicapped: A Bibliography 
of ERIC Documrents , Paul E, Schroeder- Columbus i The 
Center for Vocational and Technical Education, The Ohio 
State University, August 1973, 33 pp* (ED 083 480) 

This annotated bibliography, limited to abstracts from 
Resources in Education (RIE) , consists of documents with 
"career education," "job training," "manpower development," 
"vocational education," or "vocational retraininq" 
as the major descriptor. A selection of 37 judged 
most relevant is included in abstract form according to 
publication dates from 1970-1972, 

* * * 

* * * 
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Vocational EducatlQn for the Handicapped; A Review . 

Information Series No. 119. Marc E. Hull. Columbuas 
The Center for Vocational Education, The Ohio State 
University, 1977, 64 pp. (ED 149 188) 

This paper reviews the role vocational education has played 
in the career development of the handicapped and discusses 
the ejcpanded role it must assume. Specific topics are dis- 
cussed s "A rationale" for the participation of the handi- 
capped in vocational education and barriers to their 
participation? the impact of legislation on vocational edu- 
cation for the handicapped (P.L. 94-142, 94-482, and 93-112); 
identifying the handicapped; developing program alternatives; 
pre-vocational education; inter-agency cooperation; instruc- 
tional raaterialsi personnel preparation; and program 
evaluation 
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youthi T ransition to Adulthood (Report of the Panel on 
Youth of the President' s Science Advisory Coitmiittee) . 
James S. Coleman, chairman. Chicago j university of 
Chicago Press, 1974, 193 pp. 

The stated purposes of this report are "to examine the con- 
texts that now sKist for youth, within which they come to 
adulthood; to assess the fitness of these contexts for the 
accomplishment of the developments necessary to full 
maturity; and then to propose alternative settings that _^ seem 
to be preferable ways of accomplishing that assignment." 

Part 2 of the report details the history of age grouping of 
young people in America, rights of children, demographic 
characteristics and economic problems of youth, and current 
status of educational institutions. Issues discussed in 
Part 3 include various aspects of age segregation in school 
and in the community, the scope of formal school, and the 
legal status and rights of youth. 

Historically, treatment of youth in American society is viewed 
as having consisted of two phases— the work phase of the. 
earlier agrarian society and the schooling phase of modern 
society, wherein young people have been kept in school as 
long as oosaible and, consequently, out of the labor force. 
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mong the proposala for changing to a third phase that would 
include, but not be limited to, schooling are the followingi 

• Changing the school structure 

• Mternating school and work 

• Adding edueation of youth to the production 
function of work organisations 

• Balancing protection and opportunity rights 
of young people 

• Providing young people with opportunities 
for public service 

• Researching existing institutions and the 
effects of various environments on youth 



* * * 

* * * 
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PREFACE 



This thesaurus of vocational education outcome questions 
is one of the products of the first year of study on the Pro- 
ject "Examining Vocational Education Out -mes and Their Cor- 
relates" undertaken at the National Cent for Research _ m 
vocational Education. ^ The long-term goals o£ this proDect 
are I (a) the development of a set of outcome criteria which 
are appropriate for evaluating vocational education programs,' 
and (b) the identification of program characteristics and 
other factors significantly correlated to these outcomes. 

in general, the purpose of the thesaurus is to provide 
the vocational education community and other relevant aud,.'.- 
ences with a comprehensive, organized listing of vooationa i 
outcome questions and hypotheses. These were obtained by 
exploring the literature and through discussions with parties 
interested in vocational education programs. The functions 
and possible uses of the thesaurus will .be describsd in more 
detail in the Introduction which also discusses, some ut the 
substantive and stylistic problems that arise m developing 
a product of this nature. Moreover, Section I includes a 
discussion of the importance as well as problems of attempt- 
- ing to develop a classification system for categorizing voca- 
tional outcomes. It also explains how we have classified 
outcomes and how this organization of the thesaurus may oe 
used by a reader locate outcome questions of interest. 
Section II lists ' outcome questions, categorized according 
to education levp nd other considerations. This Compendium 
of Vocational Ed tion Outcome Questions is followed by an 
index of specif i Tot identifying some outcome questions 
that relate to a particular; topic. 

December 197 8 Joanne Farley 



-•■Other products of this study include "Examining Voca- 
tional Education Outcomes i perspective on the State of the 
Art" and "Vocational Education Outcome Studies s An Evalua- 
tive Annotated Bibliography Including an Appendix of Key Re- 
lated Literature." These documents also appear as attach- 
ments in the present report. 
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I. INTRODUCTION AND EXPLANATORY NOTE^^ 



Definition and Purpoae of the Thesaurus 

Vocational education outcomes are broadly defined as the 
consequences of vocational programs* These consequences in^ 
elude the results of (a) a student's participation in -a vocix- 
tional prograin, and (b) the existence of ongoing vocational 
programs in the conttnunity. Outcomes may affect an individual 
student, institutions, society as a whole, or some particular 
segment of society such as taKpayers, employers, or members 
of local conununities* Vocationai progrMis may be at the sec-- 
ondary or postsecondary level i they include programs for adults 
and out-of -school youth* All results, consequences, impacts, 
or effects of vocational education are considered outcomes 
whether they are positive or negative in nature/ intended or 
unintended, short-term or long-- term, economic or noneconomic. 

The Oxford English Dictionary defines a thesaurus as "a 
treasury or storehouse of knowledge as a dictionary, encyclo* 
pedia, or the like*" Tht^ authors of a recently published 
Evaluation Thesaurus suggest that "Somewhere between a die- 
tionary or glossary and an encyclopedia there is occasionally 
room for a thesaurus — ^'a book containing a store of words or 
inforination about a particular field or set of concepts' 
(Webster III)."^ Implicit in both definitions is the sugges-^ 
tion that some organizing principle exists in a thesaurus 
that (a) sets the boundaries for inclusion of some information 
to the eKclusion of other information, and (b) provides a 
means of access to the particular information desired by the 
reader. Given the above definitions then, a thesaurus of 
vocational education outcome questions is first, a collection 
of questions concerning the effects, results, or consequences 
that are in some way attributable to some aspect of vocational 
education* Seoondlyi. it is a compilation of outcome questions 
to which some principle of organisation applies that enables 
a reader to obtain desired information* The basis of organ^ 
ization of the thesaurus will be discussed in more detail 
below. 

Some individuals have argued that the set of vocational 
education objectives upon which many studies in evaluation 
have focused is too narrow* Other objectives, they claim, 
should also be emphasized and studied. Moreover, not all of 
the consequences of vocational programs are anticipated or 



■Michaal Scriven and Jane Roth, Evaluation Thesaurus , 
(Pt^ Reyes, Calif.: Edgepress, 1977) , Introduction. 
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identified in advanGe as objectives. In att^pting to form- 
ulate and/or evaluate vocational education policy^ policy- 
makers and researchers need to know more about the wide range 
and diversity of effects of vocational education programs, 
regardless of whether those effects are related to intended 
goals or are unintended by-products of vocational education* 
Hence, it is most important that some kind of identification 
and articulation of possible vocational eduGation outcomes 
be developed- A thesaurus of vocational education outcome 
questions can help in meeting this need. Ttore specifically, 
a thesaurus of vocational education outcome questions can 
serve the following functions s 

1, It can provide policy-makers and others with a com- 
prehensive (but not exhaustive) listing of the pos-- 
sible outcomes that have been specified by persons 
closely related with vocational education through 
participation and/or research. 

2, By providing policy-makers with information about 
the range and diversity of possible outcomes, the 
thesaurus can help them in their consideration of 
the goals that will be treated as desirable "given" 
consequences of vocational education policy t 

3, Related to these functions is the role the thesaurus 
can serve in identifying outcomes for policy-^makers' 
and others that have qone unnoticed because they 
were not intended. A good deal of discussion has 
revolved around the question of how effective voca-- 
tional education has been in achieving stated goals. 
However, it can be assumed that aside from the pos- 
sible occurr^^nce of outcomes that are offiaially 
deemed desirable ^ a host of other outcomes occur 
that are not anticipated or even recognized. Further 
more, these outcomes may themselves result in "spill^ 
over" effects of various kinds-^-some of which may 
have significant results that need attaiition. Hence, 
recognition and articulation of hitherto unnoticed 
outcomes could have important policy consequences. 
Such questions as (1) which outcomes should be in- 
cluded in the set of "desirable and intended" out- 
comes and (2) what intended outcomes are to be given 
higher priority than others ^ cannot be rationally 
considered without taking into account the full 
range of outcomes that may possibly occur. 



4. In terms of identifying those vocational education 
outcomes that possibly occur, this thesaurus can 
provide a valuable source of Important and interestinc 
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researGh topics for investigators of vocational 
programa* As much the thesaurus can serve as a 
fruitful starting point from which to begin form- 
ulating and testing h^fpothesei relevant to vocational 
education processes and outcomes, 

5. Moreover, the thesaurus can be used to develop 
checklists of those outcomes that may have actually 
been empirically studied and those that have never 
been seriously Investigated, 

6. For purposes of studies like that being undertaken 
by this RS^D project, the thesaurus can provide a 
collection of varied outcome questions from which 
investigators can select not only interesting out- 
comes for study but also those that merit special 
attention due to their significance for assessing 
and evaluating vocational education. This is an 
important function because outcomes can serve as 
criteria or as evaluative standards beyond pre-^ 
announced progrMa objectives and can stimulate a re-- 
examination of program implCTientation strategies and 
overall policy, 

7. Finally, the thesaurus can help researchers concerned 
with particular groups (i.e., women, the physically 
handicapped, athnic minorities, etc*) by focusing 

on specific questions pertinent to such groups. 
Should vocational progranas relevant to thei^'*. groups 
be evaluated along the same lines or with same 
criteria used in evaluating programs for regular 
students? Are there soma outcomes that affect the 
members of special groups but not regular students 
or do outcomes affecting both the former and latter 
differ in the nature of their impact on and/or sig-- 
nificance to these groups? By identifying the out- 
come hypotheses applying to special groupe, the 
thesaurus can provide a, badly needed source of in- 
formation with which to address these and other 
questions* 



The Development of Outcome Statements and Questions 

In developing the thesaurus, the project staff began 
by collecting a large number of outcome statCTaents that took 
final form as outcome questions. Before discussing the rea- 
sons for reformulating the statOTients into questions, a brief 
discussion of the importance of outcome statements will be 
helpful* 
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We emphasize that outcome statements are different from 
vocational outcomea . The contrast between the two is sig- 
nificant* As defined above ^ a voGational outcome is some 
consequence, result, or effect of participation in and/or 
the eKistence of vocational education programs. Hence, an 
outcome is an occurrence that does in fact take place in the 
world. Indeed, it is quite possible that a number of voca- 
tional outcomes exist or occur which may never be perceived 
or suspected and consequently, about which statements will 
never be made* By contrast, an outcome statement is a claim 
about some possible outcome. It is a linguistic description 
of what is stated to be the case, i,e. , a description of the 
possible outcome that someone is making claims about. Out* 
coine statements may be considered either correct or incorrect 

(unlike an outcome which does or does not occur) . An indl^ 
vidua! can posit the occurrence of an outcome in an outcome 
statOTient but the claim may be wholly inaccurate, part|ally 
inaccurate, or wholly accurate, As in the case of all "hypo- 
thetical claims, statements positing the occurrence of an 
outcome may be supported by the results of empirical testing. 
The greater the number of tests that yield positive results 
in support of a hypothesis im, the greater our confidence in 
the validity or correctness of the hypothesis will bsr Never- 
theless, our acceptance of an empirically tested outcome hy^ 
pothesis will always remain tentative to some degree. This, 
then, is a thesaurus of vocational education outcome questions 

(or hypotheses) and not a thesaurus of outcomes that are 
posited as actually occurring. ^ 

There is another reason for emphasising that outcome 
statements are different from outcomes. Outcome statements 
not only att^ipt liculate and describe the hypotheaized 

outcome in qut;stion, jUt they may well pro\^ide us with other 
information as well. Typically, an outcome a tatCTient describes 
what happens to whom and how, and in some cases also indicates 
when, where , and why. At a minimiim, an outcome statement* 
must make reference to a vocational outcome, an "affected 
entity" and an "agent." An outcome statement should not be 
confused with a process or progrian description* The state^ 
ment that "vocational programs cffer courses in consmner ed-^ 
ucation" is a description of an aspect of vocational processes. 



^Hereafter, when referring to outcomes, we will in fact 
be referring to outcomes as they have appeared in an outcome 
statement, hypothesis, or question. Hence, if we state that 
"outcomes can be categorized in a nmnber of ways,'* the reader 
should interpret this to refer to h ypothesised outcomes or 
Oijtcomes that have been posited as being actual occurrences. 
Also, for a discussion of the inherently hypothetxcal nature 
of all our scientific knowlydge, see Karl R. Popper, The Log! 
of Scientif ic Discovery , rev. ed. (London^ -utchxnson and 
CO. , 19fey) , Chapter JO, 
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It does not indicate what the effect or the impact of offer- 
ing ouch courses might be. By contrast is the statement 
that "as a result of participating in a vocational program, 
students aaquire consumer education*" In the latter, a con'- 
sequence o£ the provision by vocational programs of consigner 
education ooursas is being posited. It asserts that students 
not only utilize these courses but also that students do, in 
fact, receive a type of training in these courses*. This is 
the "vrhat" of the statement, i.e., the outcome itself. 

The "affected entity" of an outcome stat^ent includes 
the individuals, groups, social institutions, etc., that are 
most immediately and directly affected by the outconie, whether 
or not the effect was intended* In the example above, the 
affected entity is the group of individual vocational students 
who take courses in consumer education. Sometimes an out- 
come statement may be interpreted as also having an indirect 
recipient of the outcome's affect insofar as the outcome will 
have spill-over effects* For eicample, it could be suggested 
that the student* s family is also affected by the training 
he/she receives in consumer education. But in terms of using 
the statement in this thesaurus, it must at least include 
reference to some directly affected entity. 

Typically, an outcome statement will also indicate the 
"agent,*' i,e,, that aspect of vocational education which is 
identified as being the primary causal factor. The aqent 
also indicates that the outcome is brought abouv isult 
of a distinctive educational treatment, i,e., u edu- 

cation, which has results for some affected en. Thm 
agent of an outcome statement addresses the qua:, u of "how" 
the outcome comes about. In the example above, tu^ agent 
identified in the statement is a "vocational program." Some- 
times the agent is made more explicit when the statement iden- 
tifies what or which aspect of vocational programs is respon- 
sible for bringing about an outcome . In this sense, then, 
the agent could be a group of vocational educators, a distinct 
vocational field, a specific level of vocational education, or 
some other aspect of the overall vocational education process. 

Sometimes an outcome stat^ent . will indicate what time 
perspective should be used in studying a particular outcome. 
In other words, the statement may suggest that the occurrence 
of the outcome is short-term and inmiediate or is long^-lasting 
and is manifested only after some lengthy time int'srval* The 

"where" of an outcome statement would include information 
such as rural or urban programs, inner-city or suburban pro^ 
grams, perhaps different regional programs, etc. In provid- 
ing the "why" behind an outcome, an outcome statement will 
sometimes provide the rationale or theoretical basis for ex- 
pecting the outcome to occur. 
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The above should dr-noiv^+^.ra te that outcome statements be- 
qln to provide the infornuh'^^n naedea in order to inquire 
Smpirically whether or no:' a€ hyFothasi^^ed outcome occurs, 
who or what it affects^- and so ntu 



Tft.^table Ou tcome Hypptheses 

.^s an outcome statement b^comee increasingly specific 
in its description^ it usuaJlj^ increasingly takes on the 
form of a "testable" hypothesis* "Testable" in this sense 
means that there are some ways of specifying under what con^ 
ditions we expect the hypothesiiec? outcome to occur. Follow*- 
ing Carl Hempel, a philosopher of science, we can say that a 
testable hypothesis is a statement that can be put into the 
following explicitly conditional formi If conditions of kind 
C are realized, then an outcome of kind 0 will occur. More* 
over^ the statoment is such that we can empirically check to 
see if 0 occurs when conditions of kind C do in fact hold* 
Although we have reOTitten the outcome statOTents that are 
presented as outcome questions in this thesaurus as clearly 
and as specifically as possible, we are aware of the fact 
that many of them are "outcome hypotheses" but are not nec" 
essarily "testable hypotheses." Nevertheless i outcome state- 
ments of this latter kind were included as outcome hypotheses 
or questions bect^^^se the consistent anphasis they have 
been given by vocational educators, educational evaluators, 
and researchers suggests that they are important enough to 
retain as potential subjects of more thorough study. 

There are at least two basic reasons for the lack of a 
large number of clearly articulated and testable hypotheses 
about vocational education outcomes. First, the empirical 
investigations of vocational education outcomes that have 
been hitherto conducted have bm%n relatively narrow and 
limited in scope given the widely diverse range of claims 
that have been made about vocational education outcomes. 
This is to say only a few vocational education outcomes (us* 
ually perceived as goals or intended outcomes) have been 
©mpirically studied. Such investigations usually fall into 
the group of - "follow-up studies" of vocational students* 
EvBTi where this empirical work has been conducted, problems 
of operationaliiing, measuring ^ and testing vocational out- 
come hypotheses remate. Jor example i How does one check 
the validity of a finding that large numbers of OTployed 
vocational students have or have not entered training^related 



4por a discussion of testability and other methodologi- 
cal issues, see carl Hempel, Aspects of ScientHic Exglana- 
tion. (Mew York". The Free Press, 1965). 
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occupations? How should " training-relateaness" be defined? 
How can we define "vocational education" in such a way that 

feel confident that we are testing hypotheses about a 
distinctive educational treatment? These and other issues 
hava continued to plague investigators of vocational educa- 
tion. Thus, the range of hypotheses formulated for testing 
has been rather n«.rrow and even where the empirical testing 
of euch hypotheses has been perform|d, it is sometimes not 
clear how to interpret the results. 

Another major factor in explaining the problem of com- 
piling a comprehensive list of testable vocational outcome 
hypotheses has been the lack of attention paid to providing 
reasons and arguments supporting claims made about vocational 
outcomes. The ability to test an hypothesis, i.e., to oper- 
ationaliie and measure concepts in a significant way, is 
closely interrelated with the explanatory context from which 
that hypothesis is derived. Hence, the claim that "partic 
ipation in vocational programs causes students to become more 
alienated from themselves and society" is not very clear un- 
til we know what it means to say someone is allented. Per- 
haps we all share some kind of intuitive hunch about what we 
are saying in calling a human being alienated. However, we 
cannot test vague "hunches" any more than we can test vague 
statements. 

Insofar as the meaning and use of a concept differs be- 
tween theories (or explanations) , the indicators used to 
measure them will differ too. Since many (certainly not all) 
of the outcomes attributed to vocational education by both 
its critics and its advocates are pronouncements rather than 
explanatory conclusions, the outcome hypotheses will reflect 
this lack of context and remain difficult to test. However, 
the fact that many of these "pronouncements" are repeatedly 
asserted suggests that they tap real aoncerns and should be 
noted for future study and dev^>lopment . 

The point of this discussion is to indicate some of the 
problems involved m attempting to rtindaf a comprehensive 
list of outcome statements into tectable hypotheses. As 
noted above, one might think of-; the claims about outcomes 
collected in this thesaurus as hypotheses but not necessarily 
testable hypotheses. To emphasize that the outcome statements 
that were developed for this thesaurus are not c onfirmed 



^Por further discussion of these issues, see Attactmients 
"A" and "C" iin this report. 
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hvpotheses® or even in many cases, testable hypotheses, we 
have rewritten tham as outcome questions in the thesaurus. 
Hence, instead of listing outcome statements like the fol- 
lowing- "Participation in vocational programs at the high 
school level results in shorter job-search time for voca- 
tional students than that needod by comparable general/aca- 
demic students," we have reformulated the statement mto a 
question— "Does participation in vocational programs^at the 
high school level result in shorter job-search tm^ ^or 
vocational students^ than that needed by comparahiQ general/ 
academic students?" 

Prig In s a nd Formula t ion of O utcome Questions 

The Outcome questioj;: c wnpiled in this thesaurus v?ere 
obtained from a variety no .irces. Some were obtained dur- 
inq informal conversaticf; a .Uh individuals such as vocationa, 
educators, vocational adm; Jwuvtrators , city school superinten- 
dents, labor union representatives, business and industry 
representatives, researchers in vocational education evalua 
tion, and so on. However, most of the outcome qwostions were 
developed after exploring a number of written sources. The 
latter extend in range and scope from educational pournal 
articles to newsletters published by vocational youth organ- 
izations. 

Citations are not provided for each specific outcome 
question. There are several reasons for this. First, none 
of the outcome questions contained in this thesaurus are 
direct quotations. In many cases, the outcome question was 
arrived at by making eKplicit whnt some source merely implied 
or suggested while discussing various aspec^is of vocational 
processes. As has been indicated, in a great number of cases 
outcome statements or hypotheses were found that were so vagu 
and so general as to be almost meaningless except on an intu- 
itive llvel. These were reformulated in order to render them 



^By "confirmation" is meant obtaining expected test^re- 



sults over numerous testings. Some of the outcome statements 
included in the thesaurus have been confirmed for a particu- 
lar testing but none have been confirmed over numerous ^asea 
at we may have a high degree o£ confidence m thea 



so that we may have a nign aegree uj- wwijo, 4.wio*.w= -..-iir 
validity, 

"^Another reason that the project staff felt that reform- 
ulating the outcome statements into questions was needed was 
to ensure that no one would misinterpret these statements, 
thinking mistakenly they might be claims officially propound^ 
ed by The National Center . 
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as understandable to as wide a variety - f;. iisnces as poasl- 
ble. However, in reformulating these clt.iwa about outcomes, 
we often had to impose a more explicit interpretation on 
than and in doing so, may have changed the meaning and intent 
of the original source. Secondly, a great number of claims 
about vocational outcomes were repeatedly asserted or sug- 
gested by a number of different sources- Hence, it wasJ not 
clear which one should be cited. 

Perhaps the most significant consideration, affecting 
the decision to not cite sources for the outcome questions 
is the broader context within which this theoaurui may be 
used. Earlier, we suggested a number of different purposes 
that a thesaurus of vocational outcome statsnents can serve. 
However, all the functions potentially served by the thesaur 
us rest on the general assumption that each outcome statement 
(or question) can be treated as an hypothetical claim about 
the effects of vocational education. For whatever particular 
purposes an individual may refer to this thesaurus, its gen- 
eral usefulness derives from the fact that it is a compila- 
tion of hypothesized rel«.tions between vocational programs 
and educational outcomes vhich affect individuals, groups, 
or societal entities. 

When dealing with hypotheses (whether in a scientific 
context or a policy making context) a distinction is some- 
times made between the "logic of discovery" and the "logic 
of justification" (Popper, 1968) . The logic of discovery 
applies to the processes by which hypotheses are initially 
formulated and in this sense, discovered. There are no for- 
mal canons or rules of logic that cover exhaustively the 
procsdures of discovering hypotheses." In contradistinction 
to the logic of discovery is the logic of justification. 
Both in sientific practice and in everyday discourse, we 
can identify a number of formal procedures which can be used 
to gain acceptance that a conslusion (including that an 
hypothesis has been confirmed) has in fact bean justified or 
at least reasonably supported. In this coiifc<*xt then, we 
repeat for emphasis, this thesaurus is not a categorized list 
of confirmed or supported hypotheses about vocational educa- 
tion outcomes. Rather, it is intended to expand awareness 
of the questions and unconfirmed hypotheses that can guide 
scientific research, the evaluation of vocational programs, 
and aid the decisions that policy makers must confront while 



popper' and other philosophers of science do suggest 
that an hypotheses whose discovery was guided by working out 
of a particular theory is generally more acc-^ptable than one 
that seemingly derives from no particular substantive context 
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forming and implementing vocational education policy. Since 
the literature on vocational education and our discussions 
with informed individuala has served to stimulate our devel" 
opmant of outcome statements and questions rather than to 
provide authoritative sources i we do not think it necessary 
or appropriate to single out particular authors or indlvid-^ 
uals for citation as "discoverers of outcome hypotheses,"^ 

One publication that merits special comment bears the 
title -'Questions in Vocational Education Organized around 
a " systems-^oriented conceptual framework," the report identi'- 
fies six components associated with vocational education;" 
" (1) the context in which vocational education occurs, (2) 
the Inputs into vocational education^ (3) the proceas by 
which vocational education staff and facilities are struc-- 
tured and organized into programs to provide services, (4) 
the product or output of vocational education programs, (5) 
the i mpact " or result of vocational edu^vation/ and (6) the 
interrelation between conteKt^ inputs process, product and 



Some eKampies of possible sourc^^s for outcome questions 
or hypotheses include the following? Edwin L* Herr , ed. 
Vocational Gui dance and Hum an DevelP pTyient (Boston i Houghton 
Mifflin Co, , 1974")? James E, Wall/"ecn Vocational Education 
for Special Gr oups i The Sixth Yearb^^ok of the j^nerioan 
Vocation al Association, (Washington r D.C. s The American 
Vocational Association^ 1976); and in Attachment "A" of this 
report, in the Appendix of Key Related Literaturei see: 
Robert P, Quinn and Martha S. Baldi de Mandilovitch, Educa - 
ti on and . n^b Satief action I A Quegt ionable Payoff (Washing- 
tSa] D~C,t u, "S/ Department of Health /"Eduoati and Wei- 
St^rsta The National Institute of Education, March 1977)? Grant 
Vsi^^m, **Criteria Against Which Vocational Education Should be 
Held Accountable," in Interpr eting Outcome Measures in Voca - 
tional Educatio ns A Final Report , Floyd L, .McKinneyi Kenney 
E* Gray, "and Marie Abram (Columbus i The Center for Research 
in Vocational Education, The Ohio state University, 1979, 
250-285 pp.)/* James S* Coleman, Equalit y of Educational Op - 
por canity. Reconsider ed (Baltimore i Johns Hopkins Univer^ 
sity, 1967) ; and Leonsir^ A* Lecht, Evaluating Vocational Edu -- 
oation —Policies a nd Plans for the 1970 's (With an Annotated 
Bibliography) (New York i Praeger Publishers^ 1974), 

■^^Donald W. Drewes and others. Q uestions in Vocational 
Ed ucation I What Everyone Wants to Know and is Mot" Afraid 
to Ask , Career and Vocational EdUGation Professional Devel^ 
opment. Report no. 18 (Raleigh s Center for Occupational Edu- 
cation, North Carolina State University, 197 5), 
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impact;'* QuQstioni concerning vocational outcomes fall into 
the "product" and "fjyr^^at" categories of this framework. 
The thesaurue, on the other hand^ is wholly concerned with 
preeanting questions i.bout vocational Qutccmes* Some of the 
questions may have indirect or suggestive implications for 
studying processes or inputs into vocational education, Bwt 
the thesaurus is limited to presenting a comprehensive list 
of questlona about vocational education outcomes , 

There is a second significant difference between the QVE 
report and the thesaurus. The questions formulated in the 
report are oriented to obtaining descriptive information, for 
exainpler "What is the total number of program completers by 
type of school?" By contrast, the outcome questions in this 
thesaurus are oriented toward obtainira causal knowledge 
about vocational outcomes. That im m i.'ay, that most of the 
outcome questions in this manv^ s^rip^: ^'U^k to ask whether 
vocational education as a dl^^ :tivv educational treatment 
can be oausally linked to pai :i. ;.4lar effects. An example 
of a causally oriented question is "Are vocational programs 
more successful in reducing the student dropout rate than 
general/academic programs? " 



The Classification of Outcom.e Questions 

There have been various attempts made to develop a 
classification framework for organiiing and categorizing 
questions and/or statements about general educational out-^ 
comes and vocational outcomes, ^-^ However, most of these 
attempts have dealt with anticipated outcomes, dssired out- 
comes ^ or outcome goals and objectives. As was emphasized 
above, there is a need for researchers and policy makers to 
learn more about the whole range of vocational education out 
comes. This includes studying both intended and unintended, 
desirable and non-desirable, long t©rm and short term, eco- 
nomic and non-economic outcomes. 



See, for example, Richard Derr. A Taxonomy of 
SoGial Purposes of Public Schools . (New YorkT Da^Kd McKay 
C0.7 Inc*, 1973)." Also, see Morals Coburn* C : : m', 
and Salina Mushkin, Indicators of E ducational . y^' . rail 
1972 , National Center Tor Education St:atistlcs ' ton, 
D.cVi iJi S. Goverraient Printing Office, 1973); anrt ^^aw id R. 
Krathwohl, Benjamin Bloom , and Bertram B. Maijia, Taxonomy 
of Educational Objectives 1 The Classification of Educa tional 
GoalB/ Handbookll I A ffective PCTain (New Yorks -^^-^^ 
McKay Co., Inc., 1964). — ~ 



Although we have used a claasif ication system to organ'- 
Iza the outGonie questions in tha thesaurus^ its davelopmartt 
warn guided mora by pragmatic considerations than by substan- 
tive or theoretical criteria. Before proeeeding to a de« 
acription of this fraitiework, it is appropriate to consicler 
the rationale and implications of any classification system. 
The importance of such a syatOTi derives from the role that 
it serves in contributing to the efforts of voaational edu- 
cation researchers^ administrators^ and policy^makers* Fol- 
lowing are a few qf the useful functions a cLassif ication 
framework could perform, 

1* A classification sch^e would emphasize to other 
researchers or interested parties that there is a 
range o£ diverse outcomes ^that need attention in 
important areas of concern to the vogational edu* 
cation coiranunity. 

2. The development of a frameworlc upon which the class" 
ification of outcome statements is based could serve 
as a theoretical (or eKplanatory) basis for idanti- 
fying and specifying those faators that may be 
causally important in producing specific vocational 
education outcomes, 

3. A classification scheme would provide the basis for 
cumulating and organizing the results of evaluative 
studies and for identifying directions for further 
efforts. 

4. A classification schCTne would allow the researcher 
to more easily (and perhaps more senaitively) com-- 
pare and contrast outcome statements within and 
across categories. 

As the list above shows ^ efforts to develop an adequate 
classification scheme should be intensified. However^ de* 
spite the potential benefits, there are obstacles to develop- 
ing an appropriate classification framework* 

The imposition of a classification scheme or conceptual 
framework upon a set of outcome statements will impose a par- 
ticular perspective or interpretation on each of the state- 
mants* Hence/ looking at the classification schmie used by 
Bregman and Prey {sea Figure 1) one finds that vocational 
outcomes are interpreted as being in a major way a result of 
curriculum content, Also^ it can be noted that the range 
of outcomes that can usefully be categorized in thii class-- 
irie^Uon ^it^ tho^^ thftt deal with individual-level skills^ 
eompetencies^ and so on* In other words i there is no rooin 
Cor categorizing outcomes that are thought to affect 
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Clasiifying Degired Vocatioiial Outcomis in Relation 
to Individual Competencies and Character Cultivation 
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Ralph Bregmaii and Verna Prey, "Vocational Educationi An AlternatLve 
Rehabilitation Mode for Correctional Institutions" in VQcatigrial Bduca- 
tion for Spscial Groups i Sixth yearbook of the American VQcational 
Asseeiat i o L James E. Wall, edV (Washington, B.C.; The Merican 7oca» 
tional Association , 1976^ p. 170). 



inatitutioris or funational characteristica of the soGiety 
at large. Hence, the classification framework lugfested 
in Figure 1, has an individualistic £oa\xB anfl aurriculum 
bias, and outGOines or outcQme statements are likely to be 
interpreted on that baais. 

The way in which "slice up the v/crld" in a alaaai* 
fication scheme has other important ittiplioations. In effeOtf 
it demarcates or delimits those aspedts of the subject mat- 
ter that are soiftehow jud%ed to be "irtpprtant" or "approprl- 
ate*" To refer again to an example, the "matrix of expected 
outcomes" suggested by Jerome Mobs, 3x^ , is multi ^dimensional 
(sea Figure 2) . Only two of the dimeiislons need to be die* 
cussed here. One axis of the matrix d^narcates the relevant 
targets (or affected entities) to be considered^ i*e*i it 
dsmarcatea who or what needs to be taken into consideration 
in studying vQcational outcomes^ In this case. Moss suggests 
that we utilize two categories — "Students" and "Otlieri" Moss 
explained the distinction and categories by stating thats 

"A second axis^ for target, distinguishes betv^een 
axpacted student (and former student) outcomes , 
and indirect, secondary, or feedback outcomes 
anticipated in other people, agencies, or insti- 
tutions J' 1 2 

Note that in establishing these conceptual categories 
the way he haS; Moss has developed a claesif ication that 
implies that outcomes directly affecting students are of 
aonsiderahle importance but not those directly aff eating 
peraons^ Institutions, etc. ^ that fall irto the category of 
"Other," Indeed, outcomes that may be conceptualised as ^ 
directly affecting non^-students , institutions, etc*, are 
excluded from this classification framework. Moreover, .the 
fact that so many potential targets are lumped together in 
the category "other*' while "Students'* are clearly dtsting^ 
uished demonstrates the priority that the classif ication 
tendB to give to outcomes affecting students* 

Along a second axis. Moss uses three categories to dis^ 
tinguish possible outcome types^ — "Eduaational^" "Psycho- 
social;" and "Economic." Insofar as Inm has included these 
particular categories while excluding others (e.g.. Politi- 
cal), he has built into his matrix distinctions that d^ar- 
cate the types of outcomes that are relevant and appropriate 
for study. 



l%eronie Moss ^ Jr., The Eval uation of Occupational Edu- 
cation ProgrMfls, Technical Report. (MinneapQlisr Research 
Cobrdinatlng Unit in Occupational Education, Onivarsity of 
Minnesota, SeptCTiberi 1968), p, 10. 
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MatriK of EKpected Outcomes 
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tn attempting to understand the implioations of saying 
that oiia outcome has "relevanoa" or "importance" while an- 
other may not, one almost inevitably begins to equate theae 
terms with desirability, Indeed, Moss had quite explicitly 
developed his matrix on the basis of desirable and hence, 
relevant categorias. LikewisQ, in reviewing the classifica- 
tion scheme in Figure 1, we see that the hypothesiaed rela- 
tions betv^een curriculum content and types of outcomes re- 
s-ult in a "Competent Socialized Being." It is clear that 
Bregman and Prey are making conceptual distinctions in terms 
of what will contribute to the development of their idea of 
what eonatitutes a competent, aocialiaed individual. 

tndeod, most of the classif ication frameworkB developed 
for dealing with outcomes have been developed with the ex- 
plieit purpose of dealing with desired and/or expected out- 
comes. It has been suggested throughout this discussion that 
deeisions conoerning which outcomes are relevant for consid- 
eration as goals and as evaluation criteria will be improved 
if an expanded awareness of the range and diversity of out- 
comes is developed. This suggests that a classification 
framework is needed that is capable of conceptually organ- 
izing all kinds o£ outcomes and not just those that are deemed 
as desirable and/or expected. Figure 3 is one of the few 
examples we found that can do this. To be sure, such a 
framework would be establishing distinctions of relevance. 
However, the force of "relevance" here becomes that of con- 
cern because it makes a differene© in some way to human be- 
ings. 'Whether the difference an outcome raakes is judged 
desirable or undesirable will depend on the particular use 
to which the frainework is put. In any case, relevance in ^ 
this sense cannot simply be equated with goals, or the "good, 
or the "desirable." 

Because of the problems and the seriousness of the im- 
plications, we have chosen to organize the thesaurus around 
a classification system which has been developed with the^ 
pragmatic oonsideration in mind of how to provide users with 
convenient access to outcome questions of interest. Hence, 
the next section will explain how the thesaurus has been 
organized and how it might be used. 



ITse of the Thesaurus 

With a compendium ofl over two hundred outcome questions, 
it was felt that the reader could easily be overwhelmed in 
attempting to read through the list. Hence, a method of 
breaking down the questions into manageable sub-sets seemed 
desirable. The classification system- chosen to enable us 
to do this is by no means a reflection of the project's con 
ceptua.1 or theoretical assumptions although, as discussed 
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above^ it is not free of suggesting ways in which outcomes 
might be studiad. However^ emphasis should be placed on the 
fact that it was chosen more for its practical utility than 
for any research-directing implications it might have. With 
that caveat in mind, an outline in the classification is pro- 
vided below* 

A. Outcome Questions Having the Individual as Affected Entity 

(1) Secondary School Level 

(a) Related Directly to Economic or Occupational Roles 

(b) Related Predominantly to Non-^economic Roles 

(2) Postsecondary School Level^^ 

(a) Related Directly to Economic or Occupational Roles 

(3) Not Differentiated by School Level 

(a) Related Directly to Economic or Occupational Roles 
^b) Related Predominantly to Non-economic Roles 

B* Outcome QueBtions Having Social Entities or Society-in- 
General as Affected Entities. 14 

(1) Related Directly to Economic Sphere 

(2) Related Predominantly to Non-economic Sphere 

C. Outcome QuestionB Having Special Needs Subpopulations (or 
Members Thereof) as Affected Etntities 

(1) Related Directly to Economic or Occupational Roles 

(2) Related Predominantly to Non--economic Roles 

This system differentiates outcome questions in basic- 
ally three ways. First, it diatinguishes the outcome questions 
with respect to the affected entity of the outcome, i.e., the 
individuals, groups, institutions^ or social entities that 



A category for outcome questions related predominantly 
to no^n^-economic roles should logically proceed A. 2. a. It has 
been omitted because we have no entries for it in our sample 
of outcome questions. The same holds true for the omission 
in this framework of categories for adult vocational programs 
or programs for out-of -school youths. 

-^Differentiation by school levels should be included as 
categories but have been omitted due to lack of entries in 
our sample* This is also true of both major categories con-- 
taining questions that have special needs groups as the 
affected entities. 
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are most directly affected hy the outcome in question. Fol- 
lowing are the distirictions made with respect to the affected 
object, Firsti there are those outcome questions having the 
individual as affaated entity. These questions might refer 
to former or praaent studentSi parentSi employars, etct The 
point, hovrevar, is that the loous of affect of the outcome is 
the individual. Secondly, there are outcome questions having 
soaial entities or society-^in-general as the affected entity. 
Instances of this are when institutions # local coitmunities , or 
organized groups are being posited as the locus of the outcome's 
effects^ Finally^ there are outcome questions having partic- 
ular relevance for special needs subpopulationa as inoluding 
the following groups i blacks i the disadvantaged, ethnic and/ 
or linguistic minorities, out-^of -school youths # women, native 
Ameriaans, older vrorkers, ex-offenders, migrants, and the 
mentally or emotionally or physically handicapped* It should 
be noted that the outcome questions having the individual as 
the affeeted entity are not mutually exclusive from those . 
having manbers of special needs groups as the affected entity. 
We have distinguished between the two on the practical basis 
that many resaarcherij and policy makers are interested in 
outcomes affacting individuals as members of special needs 
aubpopulationa and not as members of the general public. 

h second method of differentiating outcome questions is 
based on differendes in school level, i.e^^ secondary and 
postsecondary* tt has been suggested that differences in 
school level may be associated with differences in the nature 
and frequency of occurrence of vocational outcomes. This 
may or may not be the case. However, we distinguished ques-- 
tions by school level because for various purposes, individ- 
uals desire or need to inquire into vocational programs at 
different school levels. Thus, where one person might be 
solely interested in evaluating outcomes at the secondary 
school level, another might be focusing on adult vocational 
progrMas. 

The third distinGtion utilized in the classif ication 
systCTi is perhaps the most likely to cause disagreCTient or 
unease among readers. This is the distinction between those 
outcome questions which we have labeled as economically or 
occupationally related and those labeled noneconomicaliy 
related. A^lthough it is always difficult to separate the 
"economic" from the "social," "political," and even "psy-^ 
chological, aspects of human life, we have tried to resolve 
this problem by applying the distinction in a fairly narrow 
manner. "Economic or occupational-related roles or spheres" 
should be interpreted in the thesaurus to apply to those 
outcome questions which are eKplicitly related to CTiploya- 
bility, ^ployment, wage rates, economic resource levels, 
and so on. However, an outcome that on the face of it seems 
to be obviously economic in nature, e,g,, employment/unemployment 
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rates for vocational graduates, may have serious soeial and 
politiQal implications as welli Hence ^ the reader is fore- 
warned against assuming that the way in which we have dis- 
tinguished between economic and noneconomic outcome ques-^ 
tione is meant to be in any way definitive • Again , this dis- 
tinction was ueed mainly becauie it seemed to provide the 
practical advantage of categoriging the outcome questions in 
an accessible manner for readers* 

Having differentiated the outcorne questions on the basis 
of affected entity, school levels and whether the outcome 
is economic or non-^economic in nature, the resulting organ** 
Ization of the outcome queBtions is made more manageable. 

Moreover, for those persona interested in a particular 
topic, we have provided an indaK of specific topics which 
follows the compendium of outcome questions. Although the 
list of topics and sub-headings is not exhaustive, we have 
tried to include those that are relevant to the concerns and 
interests of individuals Involved In the area of vocational 
education . 

The outcome questions in the compendium have been num** 
bered for identification sequentially, beginning with #1 and 
extending to #228* Although it is hoped that most of the 
major topics relevant to vooational education are Included 
in the index, it is not exhaustive with regard to the possi-^ 
ble topics that could have been included* Lastly, topics 
have been cross-referenced where it seemed advisable and/or 
appropriate. The major topic headings are arranged in alpha- 
betical order, followed by the identification numbers pre-^ 
ceding each outcome queetlon in the compendium. For example, 
if the earnings of vocational studentB is the' topic of inter- 
est, a reader would find "earnings" entered in the index in 
the following way* 

Earnings, of vocational students, 38, 39, 40, 88, 89, 
90, 91, 92, , . * • 

The reader can then refer to the outcome questions having the 
identification numbers listed after the topic. We have not 
exhaustively listed every outcome question potentially re- 
lated to a specific topic* In obvious cases this has been 
represented by a series of periods following the last iden- 
tification number listed. For exOTiple, given the large num- 
ber of questions dealing with "postsecondary" students, "post- 
secondary has been entered as belowi 

Postsecondary education, vocational, 30, 31, 32, 33, 

34, « * . . 

The reader is encouraged to scan the list to insure that all 
outcome questions relevant to the topic of interest have been 
noticed. 
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II. COMPENDITIfl OP VOCATIONAL EDUCATION OUTCOME QUESTIONS 



A. OUTCOME QUESTIONS HAVING THE INDIVIDUAL AS THE AFFECTED 
ENTITY I ' • ■ 

(1) SEGONDARY SCHOOL LEVEL; ' ' . 

(a) DIRECTLY RELATED TO ECONOMIC OR OCCUPATIONAL 
ROLE 

1. Outcome nuestlon i Does participation in vocational 
programs at the high school level result in a satis-^ 
factory number of the graduates obtaining their first 
full-time employment in training-related fields? 

2. Outcome Question s Does participation in vocational 
programs at the high school level instill the belief 
in students whose first job is training-related that 
they have been well prepared for their jobs? 

3 i Oat come Queetion i Are the returns from individual 
investments in vocational education higher than re- 
turns made by individuals in general/academic pro- 
grams below the baccalaureate level?! 

4, Outcome Question : Does participation in Distributive 
or~Trade" and Industry programs at the high school 
level result in more students having jobs lined up 
for starting work immediately after program completion 
than dg comparable students in other vocational pro-- 
grams?" 



When a contrast group is referred to in an outcome ques^ ^ 
tion like the above, it is assumed that the comparison is be^ 
ing made for groups whose members are matched in sex^ age, 
mental ability and socioeconomic background* 

2- 

"In a number of the outcome questions^ the reasoning be- 
hind citing a specific vocational program (or a specific time 
period as in question #8) will not be obvious* Nevertheless, 
we have retained these formulations of the questions as a way 
of suggesting that new outcomes might be identified by think^ 
ing in terms of the outcomes that are possibly associated with 
different programs* Moreover ^ the different time perspectives 
we associate with hypothesised outcomes may possibly generate 
new ideas and outcomes* (Refer to discussion in Introduction,) 
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5» Outcome QueBtion i Does graduation from vocational 
programs at the high school level result in lower 
rates of unemployinent for students than those for 
general/academio high school graduates? 

6i Outcome Questionj Do graduates of vocational programs 
encounter fewer periode of unemployment than graduates 
of general/academic high school programs? 

7. Outcome Question i Does graduation from a vocational 
program" at the high school level result in briefer 
periods of unemployment than graduation from general/ 
academic programs? 

8* Outcome Queetiori i Does participation in Health pro- 
grams at the high school and community college level 
result in program completers spending more time than 
completers of other programs at comparable school 
levels in full--tinie^ permanent employment during the 
first three years after program completion? 

9p O utcome Question i Do graduates of Technical or Trade 
cjnrj Industry programs at the high school level express 
*j/tcyifc,/ d 3 a f *^^:M ^/j with th<^ir first full-time job 

same level? 

10. Outcome Question * Do graduates of Agricultural pro-^ 
grams at the Tiigh school level obtain jobs with lower 
socioeconomic status than graduates of all other voca- 
tional program areas at all other school levels? 

A. OUTCOME QUESTIONS HAVING THE INDIVIDUAL AS THE AFFECTED 
ENTITY ; 

(1) SECONDARY SCHOOL LEVELS 

(b) RELATED PREPOMINANTLY TO NON-ECONOMIC ROLES 

11, Outcome Question s Does participation in vocational 
programs and affiliated youth organizations result 
in students having experiences that enhance their 
ability to respond^appropriately to a variety of 
social situations? 



When an outcome question does not eKplicitly refer to 
the secondary school level but does refer to "youths" or 
"youth organizations" affiliated with vocational programs # 
we have categorized it under secondary school level. 
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12, OutCQme Questioyi i Does participation in youth organ- 
i nations develop iri students leadership capabilities 
in planning and Implementing small-group activities? 

13, Outaome QuaBtlon i Does participation in vocational 
programs and affiliated youth organizations develop 
in students the capacity to be self -directed and re- 
sponsible in pursuing both occupational and non-occu- 
pational goals? 

14, Outcome Question s Does participation in vocational 
organizations develop in students a personal inter^ 
est in affairs extending beyond the confines of the 
local community? 

15, Outcome Questio ns Does participation in vocational 
programs and affiliated youth organizations develop 
in youths a sense of responsibility for the welfare 
of others? 

16, Outcome Question s Does participition in vocational 
programs and affiliated youth organizations motivate 
and prepare students to enter into voluntary^ social 
service activitiea more so than does participation 
in general/academic programs by comparable students? 

17, Outcome Question s By providing some sort of finan-- 
cial incomei So cooperative vocational progranis de-^ 
velop the studenhi ' independence and ability to assume 
responsibility outside the parental environment? 

18* Outcome Queition i Does participation in vocational 
education and affiliated youth organiisations develop 
in students the personal qualities needed for suc- 
cessful work entry and the aseumption of responsibil^ 
ity as citizens and family members? 

19, Outcome Question t Does participation in activities 
by vocational programs and affiliated youth organi- 
zations enable young persons to establish an appro- 
priate balance between work and leisure and to select 
leisure activities appropriate to their life styles? 

20. Outcome Question i Does participation in vocational 
programs" make the emotional and psychological tran-- 
aition from adolescence to adulthood easier for the 
student than participation in general/academic pro- 
grams? 
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21. Outcome Que at ion ; Have residential vocational 
schools provided youths with new opportunities by 
rOTioving many of them frpni the care of unfits ujiin- 
terested parents and from overcrowded r impoverished 
neighborhoods where the temptation of delinquency 
and crime is strong? ; 

22. : Outcome question : Are vocational programs at the 

high school level more successful in reducing the 
student dropout rate to a lovi?er level than that 
found in general/academic high schools without voca- 
tional programB? 

23. Outcome Question i Are atuderits who participate in 
vocational programs at the high school and postsec- 
ondary level more perslBteiit- in their desire to com- 
plete their programs of gtudy than are those enrolled 
in general/academic programs? 

24. Outcome nuestion i Does partlGipation in vocational 
programB at the high school level create a desire in 
students for postsecondary BChoolingr especially in 
a field of study parallel to their high school pro- 
grams? 

25. Outcome QueBtion i Do Btudants of vocational programs 
at the" secondary level evaluate their overall high 
school eKperience more positively than do comparable 
general/academic studentg? 

26. Outcomie question s Does graduation from a vocational 
program at "the high school level cause students to 
be rejected for entry into inany two-^year, postsecon- 
dary institutions on the basis of unacceptable credit! 

27. Outcome question s Does graduation from a vocational 
program at the high school level result in students 
having insufficient or non-acceptable credits for 
admission to a four-^year aQademic/general college? 

28. Outcome Question s Does participation in vocational 
programs at tlie high school level motivate graduates 
to enroll in postsecondary programs (e.g.^ colleges, 
universities^ community colleges^ vocational-^tech- 
nical schools, etc.)? 



^Outcome questions that pertain to both the secondary and 
postsecondary school levels have been categorized under sec- 
ondary school level and are not listed again. 
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29. Qutconie Question i Does graduation from vocational 
high schools enable students to be admitted to many 
community colleges without taking "challenge eKams"? 

OUTCOME QUESTION S HAVING THE INDIVIDUAL AS THE EFFECTED 
ENTITY? ' ~ ^ ~ ^~ 

(2) POSTSECONDARY SCHOOL LEVEL f 

(a) RELATED DIRECTLY TO ECONOMIC OR OCCUPATIONAL 
ROLES V 

30. Outcome Question i Do students participating in voca^ 
tional programs at the conununity college level rate 
the quality of their vocational instruction higher 
than the instruction in the academic subjects which 
are a part of their curriculum? 

31. Outcome" Question : Do graduates of vocational pro- 
grams at the postsecondary^ non=college level rate 
their instruction higher in quality than graduates 
of high school vocational programs? 

32* Outcome Question i Do students, in their first train-- 
ing-related jobs after completing postsecondary ^ non-- 
college vocational programs consider themselves well 
prepared for their first job? 

33. Outcome Question i Does participation in vocational 
programs at the postsecondary level lead students to 
believe they would not have gotten their first full- 
time employment without the postsecondary training? 

34* Outcome Question : Do students ^ after completing a 

post> secondary vocational program^ spend mora of the 
subsequent three years in full-time employment than 
those who complete high school vocational programs? 

35, Outcome Question i Does graduation from vocational 
programs at the postsecondary ^ non-college level re- 
sult in lower rates of employment for graduates for 
the first three years after graduation than those for 
graduates of programs at other levels? 

36. Outcome Question i Does participation in Agricultural, 
Technical, or Health programs at the corraiunlty college 
and postsecondary/ non'-college levels contribute to 
the student's ability to retain a job received prior 
to graduation? 
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'37. OatCQroe Question i Does participatiori in vocational 
progframs at the" postBecondary , rion--aollege level xb^ 
suit in a decline in the proportion of graduates 
holding training-^related jobs three years after grad- 
uation? 

38 • OutCQme QueBtiQn i Does graduation from a vocational 
prograjn at the postsecondary level ensure that grad- 
uates will receive higher hourly wage rates on their 
first job than those reoeived by high school voca- 
tional graduates? 

39, Outcome Question i Does graduation from a vocational 
progr¥m at the community college level lead to better 
increases in hourly wage rates for graduates in the 
first three years after graduation than for graduatss 
of postsecondary^ nan-^vocational programs? 

40, Outcome Question i Does graduatioh froin vocational 
programs at the postsecondary # non-college level 
generally provide graduates with advantages (e.g*/ 
higher paying jobs, more prestigious jobs, etc) 
over students graduating from high school vocational 
programs? 

41, outcome Question i Do graduates of vocational programs 
at the cortmiunity collegel level obtain better jobs 
with higher socioeconomic status within three years 

of graduation than those obtained by graduates of 
postsecondary I non-^college programs and by high 
school graduates? 

42* Outcome Question i Does graduation from Agricultural 
programs at the postsecondary ^ non^college level lead 
to jobs with higher socioeconomic status than jobs 
obtained by Agricultural program graduates at the 
high school and comrnunity college levels? 

43, Outcome Question i In general ^ does participation in 
vocational programs at the community college level 
contribute to greater job satisfaction among Miployed 
graduates than that found among graduates of post- 
secondary, non--college and high school programs? 

44. Outcome Question s Does participation in vocational 
prdgr^s at the community college level contribute 
to greater job disiatisf action among male graduates 
than among female graduates at the same level? 
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45, outooma Question i Does participation in male^domi- 
nated vocational programs (e.g., Trade and Industry, 
Distributive^ or Technical) at the Gomaunity college 
level contribute to greater job dissatisfaction 
ainong female graduates^ than that found among male 
graduates of the same program? 

46. Outcome Question i Does participatiDn in Distributive 
programs at the postsecondary ^ non-college level con- 
tribute to greater job dissatisfaction among employed 
graduates than that found among employed graduates of 
other vocational programs at the same level? 

A. OUTCOME QUESTIONS HAVING THE INDIVIDUAL THE AFFECTED 
ENTITVy ~ 

(3) NOT DIFFERENTIATSD BY SCHOOL LEVEL ^ 

(a) RELATED DIRECTLY TO ECONOMIC OR OCCUPATIOmL 
ROLES ~ " . 

47 i Outcome Question : Does participation in v^ocational 
programs" produce students who are certifiable as 
occupationally proficient and ready for antry--level 
employment upon program completion? 

48. Outcome Question i Does participation in v^ocational 
programs result in etudents who are "late program 
leavers" but not program completers receiv^ing mar- 
Jcetable skills which enhance their ^ployability? 

49, Outcome Question i Does participation in vocational 
programs result in etudents acquiring psychomotor 
skillsi e.g.^ hammering, typing, and plowing, which 
ara necessary for many fornis of occupational compe-^ 
tence? 

50* Outcome Question i Doas participation im \rocational 
programE enable students to learn amployment skills 
anfl basic work disciplines in a more effective man- 
ner than persons who learn such skills from on-the-^ 
job training in industry? 

51. Outcome Question s Of all vocational program gradu^ 
ates^ do those trained in Health programs receive 
the highest proportion of training-^relatad jobs as 
first employment? 

52. Outcome Question ! Does participation in Agriculture 
programs as compared to other vooational programs 
result in the smallest proportion obtaining their 
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first full-time employment in training-reiated 
fields? 

53, OutGome Quest ion i Are students trained in vocatioiial 
prograrns bettar able to apply the diverse occupational 
skills Isarned in school than graduates o£ genaral/ 
aGademic schobls? 

54, OutaQme Question i As a result of participation in 
vocational programs i do students acquire safe work- 
ing habits and techniques i thereby reducing physical 
riekg to their co-workers and themselves? 

55, Outcome Quest ion i Does participation in vocational 
prDgrams" produce students with a substantial and 
stable set of behaviors which underly all the pro- 
fessional ^ skilled I and technical occupations? 

56* Outcome Question i Does partiGipation in vocatipnal 
education provide students with the skills, under-' 
standings^ and appreciations needed to upgrade or up- 
date their occupational competence? 

57, Outcome Question ^ Does participation in vocational 
progtaniB produce students with transferable skills ^ 
thereby increasing their chances of successful em- 
ployment even in |ields outside those in which train- 
ing was reGeived? 

58, Outcome Quest ion i Does participation in vocational 
programs produce students with skills so specific 
and speoialized that their occupational options be- 
come narrowly restricted and inflexible? 

59, Outcome Queationi Does participation in vocational 
programs prdduce students with general transferable 
skills, thus enabling them to adapt quickly and suc^ 
cessfully to changing technological and skill re-- 
quirements? 



This outcome question, like several others, has two fac^ 
ets: Do vocational students reoeive transferable skills from 
participating in vocational programs; and, if mo, do these 
skills really increase their chances of successful einployment 
in fields other than those in which they v^ere trained? Read- 
ers are not being asked to assOTie a positive answer to the first 
question and then to focus on the second question* The en- 
tire question is hypothetical, requiring further attention. 
This is true for all compound, coftpleK questions in the 
Thesaurus . 
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60 r O^tcQnte Question s Are many vocatioiial programs tied 
so telosely to local needs aiid opportuftities that the 
student's future ability to becorna ow-upationally 
mobile is limited? 

61, Qytcome Question ; Does participation in vocational 
programs produce students with training and slcills 
that will enable them to becoma geographically 
mobile? 

62, Outcome Question i Does an understanding of the aging 
prooass acquired through short Home Econoitiics cQurBes 
in gerontology prepare students for jobs which in-- 
volve working with the elderly? 

63, Outcome Question i Does participation in vocational 
programs diecourage students from obtaining training 
in areas identified as declining occupations? 

64* Outcome Question : Does participation in vocational 
programs dev-"elop in Btudenta a greater awareness of 
the occupational options available in the work world 
than the awareness held by comparable students of 
general/academic programs? 

65. Oatcome Question i Does participation in vocational 
programs enhance the students' understanding of 
their educational and career opportunities/ thiis 
enabling them to formulate more realistic goals for 
the future? 

66* Outcome Question i Do vocational aducatora and coun- 
selors impose occupational decisions on yQUths before 
they .have^ sufficient understanding and knowledge of 
occupational options on which to base their own de-^ 
eisions? 

67, Outcome Question i Does patrticipation in vocational 
programs make "students aware of the training r^guired 
for. pursuing various occupations? 

68. Qutcome Question s Does participation in vocational 
programs enable students to maka sound decisiona 
about particular occupations based on their explora- 
tions of alternative occupations in school? 

69* Outcome Question i Does the organization off many 
vocational program curricula cause students to be 
tracked into^ specific occupations? 
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70. Otatcome Question i Do the curricula of many voca- 

tional programs produce students trained in occupa- 
tions with ee\rerely limited or diminishing job oppor* 
"ttanities? 

71* Outacnia QueBtion i Do students through thei^- partic- 
ipation in vocational programs acquire skills that 
axe (or soon v/ill be) obsolete in terms of labor 
itiarket deraand? 

72. Outcome Questioni Do students , by participation in 
vocational ptaqxams, receive help from a structured, 
eareer guidance program? 

73. Outcome Quastioni Are employers now imposing higher 
edaaational requirements for jobs because vocational 
programs continue to supply a pool of trained vrorkers? 

74. Outcome Questioni Does participation in vocational 
programs create a greater awareness in students of 
the necessity of learning basic academic skills in 
becoming employable than the awareness developed in 
comparable general/academic students? 

75. Outcome Questioni Do vocational graduates usually 
resort to self job^placement because vocational 
placement offices are ineffective in finding jobs 
for students? 

76. OutcQme Questioni Does participation in vocational 
programs enable students to operate successfully in 
the labor market? 

77. Outcame Question s Do students through participation 
in vocational programs acquire employment survival 
skills which contribute to their mcomns in retain- 
ing jobs? 

78. Outcom e Question i Do students through participation 
"in cooperative "vocational programs usually enter the 
labor market with good work records in hand? 

79. Outcome Questiori :; Does participation in vocational 
prdgrama prepare students who are entering nontra-- 
ditional roles with coping competencies as well Bx/ 
the necessary occupational skills? 

80* Outcome Question s Does participation in vocational 

programs enable students to acquire job-seeking skills 
such mm how to write a resume and where to look for 
job oppDrtunities? 
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81. Ou tcc^e Ques t Ion i Does participation in vocational 
programe prepare students to learn from prospective 
ampioyers whether a job is appropriate to their skills, 
job axpectations and desires i and physical or ago- 
nomic needs? 

82. Outaome Question : Does participation in a vocational 
program enable Btudents to learn how to approaoh and 
interview proapective employers? 

83. Outcome Question i Do students entering the work force 
directly after graduation from vocational programs 

(at all levels) face shorter periods of job-search 
time than comparable students completing general/ 
academie prgramg? 

84. Outcome Question i Does participation in vocational 
programs motivate graduates to be more persistent 
in job-hunting after leaving school than general/ 
academic graduates? 

85* Outcome Question i Does participation in vocational 
programs (at all levels) result in students being 
more satisfied with their jobs two years after grad-- 
uation than comparable general/aaademic students 
holding similar jobs? 

86* Outcome Question s Does participation in Health pro- 
grams |at all levels) result in graduates better 
satisfied with their first full-^time jobs than grad- 
uates ofl other vooational programs at all levels? 

87* Outcome Question i Does participation in vocational 
programs result' in employers rating vocational stu-- 
dents as being more satisfactory as entry^level 
employees than comparable general/acadOTilG students? 

88* Outcome Question i Does graduation from vocational 
programs (regardless of school level) result in 
higher hourly wage rates for graduates in the first 
three years after graduation than graduation from 
' .general/academic programs at comparable school levels? 

89^ Outcome Question i Does graduation from an ^grieul-- 
turai. program result in a higher mean hourly wage 
rate for graduates on their first full-time job than 
graduation from other vocational programs? 

90 # Outcome Question i Does participation in vocatimal 
progranis result in greater increases in hourly wage 
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' rates for those graduates working in training-related 
areas fchaii those received by vocational graduates 
working in non-training related areas? 

91, Outcome Question i Does participation in cooperative 
progranis "in vocational education enable students who 
are forced to work for economic reasons to earn while 
receiving vocational training? 

92. Outconie Queition i Does participation in vocational 
• programs result in a decrease in earnings of voca^ 

tional graduates in the ten years following gradua- 
tion relative to the earnings of general/academic 
college graduates? ^ , : 

93* Outcome Queetion i Does graduation from a vocational 
program ~in an urban school result in higher wages 
and earningB on a student -s first full-time job than 
those received by graduatee of vocational programs 
located in rural schools? 

94. Outcome Question i Does participation in and gradua^ 
tion from Ofiice^ Technical ^ or Distributive programs 
result in jobs with a higher socioeconomic status . 
for graduates than the jobs obtained by graduates 
from other vocational programs? 

95, Outcome Question i By participating in vocational 
programs/ do program completers move to better pay-^ 
ing and more prestigious jobs (with greater oppor-^ 
tunities for advanoement) than comparable general/ 
academic program graduates? 

96* Outcome Question i Do students ^ as a result of par- 
ticipation" in vocational programs ^ become absorbed 
in progt^am activities which keep them out of the 
labor force entirely or limit their labor force par*- 
ticipation to part-time employment? 

97. Outcome Question ! Does participation in a vocational 
prdgram at any school level result in longer tenure 
for graduates on their first full-time job than for 
general/academic graduates at comparable school 
levels? 

98. Outcome Qyestion i Do vocational programs perpetuate 
the tendency of vocational educators^ administrators/ 
and society at large to regard and treat vocational 
students as human resources whose economic value can 
be enhanced by selective educational investments? 
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99- Outcoine QUestlorii Does participation in vocational 
programs tesult in students coming to regard them* 
salves as hiiman resources whose economic value can 
be increased by selective educational investments? 

loo* Outcome Question; Does vocational education foster 
parant^student discussion of the student's future 
plans (e.gt# college, postsecondary vocational train- 
ing, etc.) because it involves parents in the devel-^ 
Qpment and implementation of career guidance programs? 

A. OUTCOME QUESTIONS nhVim THE INDIVIDtJAL AS THE AFFECTED 



101. Outcome Question i Does participation. In vocational 
programs de'velop in students a greater appreciation 
for learning in general than does participation in 
general/academic programs? 

I02i Outcome Question s Does participation in vocational 
programs provide students with individualized in-^ 
struction which minimizes the learning problems 
created by differences in background and learning 
abilities? 

103. Outcome Question i Does vocational education in gen^ 
eral provide students with training lower in quality 
than that provided by college preparatory programB? 

104* Outcome Question i Do vocational students experience 
self -diminishing failure less often than do general/ 
academic students because vocational programs are 
tailored to meet the student's individual needs and 
abilities and are organi?*ed to ensure success? 

105# Outcome Question i Does participation in vocational 
prdgrams on the high school level contribute measur^ 
ably to the moral development of students? 

106 i Outcome Question s As a result of their participation 
in vocational programs at all school levels, do 
students respond well to teachers on the level of 
personal relations since they have shared interests 
in the particular vocational progrMa? 



ENTITY; 



(3) NOT DIFFERE NTIA TED BY SCHOOL LEVEL ; 



(b) RELATED PREDOMINANTLY TO NON-^ECONOMIG ROLES 
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107 • Outcome Question s Does participation in vocational 
progranis motivate students to stay in school? 

108 • OutcQinfi Quastion i Does participation in voGational 
programs result in students learning to take pride 
in their work? 

109. Outcome Question t Through participation in vocational 
programs r do students become aware of the means by 
which they can continue their learning outside of the 
formal system of schooling? 

110. Outconie Question i By participating in vocational 
prbgrams do students learn technical skills which 
provide them many consumer benefits? 

Ill* Outconte Question : Does participation in vocational 
programe afford students training in consumer edu- 
cation as part of the total instructional program? 

112. Outcome Question i Does participation in vocational 
programs enable students to become more efficient 
in their use of external resources? 

113* Outcome Question s Does participation in vocational 
programs' contribute to the student's development of 
personal economic stability? 

114. Outcome Questions Does participation in vocational 
programs develop in studenta new material wants and 
needs and in so doing/ develop heightened expecta- 
tions concerning the students' abilities to satisfy 
these wants and needs after program completion? 

115. Outcome Question ; Does participation in vocational 
programs develop in students a higher sense of per-- 
sonal growth and significance derived from working 
in occupations of their own choosing? 

116. Outcome Questions Does participation in vocational 
programs acquaint students with the scientific and 
social bases and learnings of their occupational 
pursuits such that they will not inevitably sink to 
the role of appendages to the machines they operate? 

117. Outcome Question: Do vocational program participants 
become or remain more alienated from the society in 
which they live than comparable general/acadfflfilc 
students? 
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118; Outcome Question i Does participation in vocational 
programs result in students becoming or remaining 
more alienated from themselves than do comparable 
academic students? 

119. Outcome QuestiQn ! Does participation in vocational 
programs prepare students to work independently^ 
thereby reducing the need for constant supervision 
and/or instruction? 

120. Outcome Question s Do students in vocational programs 
become increasingly responsible by executing specific 
work tasks assigned as training eKperiences? 

121. Outcome Question i Do students in vocational educa- 
tion learn teamwork and leadership skills by pursu- 
ing interdependent activities with collective goals 
as training eKperiences? 

122. Outcome Question ; Does participation in vocational 
programs eriarble students to form sound judgments 
with regard to career-related choices? 

123. Outcome Question ; Does participation in a vocational 
program lead students, through their group task 
assignments, to tolerate and appreciate differences 
in opinions and/or strategies for problem solution? 

124. Outcome Question i Does participation in a vocational 
program develop within its students a greater capac^ 
ity to organize and utilize their intellectual re-- 
sources for problem-solving than participation in a 
matched cohort of general/academic students? 

125. Outcome Question i Do students, as a result of par- 
ticipation in vocational courses based on self -^teaching 
perform batter in original and self-eKpressiva activ- 
ities, indicating facility with inventive thinking 
that explores original and/or alternative solutions 

to problems? 

126. Outcome Question i Does participating in a vocational 
program develop in students the ability to evaluate 
ideas, situations, or data in terms of meaningful 
criteria, both objective and subjective? 

127. Outcome Questlon i Does participation in vocational 
programs help students improve their communication 
skills, e.g., receiving and transmitting messages, 
receiving and interpreting information, getting a 
point across to listeners? 
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128* Outeome Question i Do vocational programs^ by locial- 
iging individuals into the mainstreani culture^ create 
and/or reinforce their desira to establish and main-' 
tain a etabla family life? 

129, Outcome Queetlon i Does participation in vocational 
progrMns "develop in students improvad self ^identity ^ 
self-esteem^ and self-confidence? 

130. Outcome Question i Do studenti through participation 
in vocational programs become steretoyped as "under^ 
achievers" or as persons who cannot succeed in a col^ 
lege-^preparatory program? 

131* Outcome Question i Do students through participation 
in vocational programs develop attitudes which lead 
them to accept some regimentation on the job? 

132* Outcome Question i Does participation in vocational 
programs lead students to develop good interpersonal 
relationships with persons of diiferent ages^ sub-^ 
cultures , and social classes? 

133. Outcome Question : Do vocational programs reinforce 
racial prejudice among students in public education? 

134. Outcome Question i Does participation in vocational 
programs develop within students the desire and 
ability to integrate their occupational pursuits 
with their cultural^ philosophical ? and intellectual 
interests? 

135. Outcome Question i Does participation in vocational 
programs iead~students to an underatanding of the 
complexity of the industrial and post-- industrial sys-- 
tern? 

136* Outcome Question ; Do students^ as a result of their 

participation in vocational programs, develop informed 
opinions about complex social, economic, and politic 
cal issues arising in industrial society? 

137, Outcome Question i Does participation in vocational 
programs lead students to participate actively in 
civic affairs and political life because of selff-- 
confidence acquired in school? 

138, Outcome Question s Do students, through participation 
in a vocational' program, develop knowladge and appre^ 
ciation of their rights and duties as a citizen? 
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139. Outcome Question s DoeB vocational education strengthen 
"parental support for particular vocational programs 

by involving the parents in the development and imple- 
mentation of career guidance prograniB? 

140. Outcome Question: Has vocational education generated 
more approval and satisfaction among parents regard-- 
Ing its curricula than have general/acadeinic schools? 



B. OUTCOME QUESTIONS HAVING SOCIAL INSTITUTIONS OR SOCIETY - 
It3-GENEM L AS THE AFFECTBD ENTITIES I 

(1) REIATED DIRECTLY TO ECONOMIC SPHERE 

141. Outcome Question i Does vocational education ^ through 
its consultative services ; help small businesses in 
urban areas remain in operation; thereby maintaining 
jobs and creating new ones? 

142 » Outcome Question i Does vocational education influ- 
ence job creation hy interacting with those develop- 
ments that create jobSf e.g*! since natural resources 
are developed when vrater drillers are trained, are 
more jobs thereby created? 

143* Outcome Question i Does the establishinent of voca- 
tional education systems provide benefits in the form 
of jobs for suppliers of equipment and job support 
services used in vocational education? 

144 # Outcome Questlon f Does vocational education benefit 
society to any significant degree by providing jobs 
for teachers y administrators, and other vocational 
staff? 

145* Outcome Question i Does the ongoing existence of 

vocational programs lead to an equitable distribu- 
tion of the nation's InGOme with earnings based on 
worker productivity? 

146, OutcQme Question : Does the establishment of a voca-- 
tional education system cause new risk capital to be 
attracted to the state, thereby strengthening the 
local economy? 

147 ■ OutcQme Question i Has the ongoing existence of voca- 
tional programs within local communities been a major 
factor in bringing new industries to the area? 
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148. Outcome QueBtlQn s Has one of the major impacts of 
vocational eduGation been the rationaligation of 
human reBOurces, thereby contributing to the rise in 
general standards of living? 

149. Outoome Question s Does the ongoing eKistence of a 
vocational program free business and industrial re- 
sources by supplying trained employees at less cost 
than that of business and industry in training a work 
force? 

150. Outcome Question i Does vocational education^ through 
lack of planning, contribute to oversupply of man- 
power in some occupational areas and undersupply in 
others? 

451. Outcome Question i Does the ongoing eKistence of a 

vocational education systKn significantly strengthen 
the manpower base of the local and state economy? 

152 i Outcome Question s Do vocational programs respond to 
and meet the emerging manpower needs at the local , 
state, and national levels? 

153. Outcome Question i Does the establishment of voca-- 
tional programs speed up industrial development at 
the local/ regional , state- and national levels? 

154. Outcome Question : Has the ongoing existence of vooa'- 
tional education improved the tradeoff between unem- 
ployment and inflation? 

155 i Outcome Question ! When there is little need to train 
youth for employment, as in a depression ^ is the ex- 
istence of vocational programs too costly and unpro-* 
ductive? 

156 i Outcome Questlon i Are vocational programs more costly 
than general/academic programs due to the use of ex- 
pensive equipment and a lower student- teacher ratio? 



OUTCOME QUESTIONS HAVING SOCIAL INSTITUTIONS OR SOCIETY - 
IN-GENERAL AS THE M^FECTED ENTITIES ; 

(2) RELATED PREDOMINANTLY TO THE NON-ECONOMIC SPHERE 

157^ Outcome Questlon i Has vocational education been an 
Important" factor in blurring the demarcation between 
socioeconomic classes, e,g## because it provides 
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individuals with skills enabling them to obtain jobs 
which# although they may be considered "working class i" 
pay wages indietinguishable from those ooneidered 
"middle claBs''? 

158* Outcome Question i Have vocational prograiTis oontinu* 
ally reinforced the stratification of groups along 
social class lines? 

159. Outcome Question i Has vocational education fostered 
curriculum differentiation resulting in ability 
grouping, testing^ and guidance counseling # which in 
turn have fostered clasB segregation within the 
schools? 

160. Outcome Ques tion > Has vocational education fostered 
curriculum differentiation resulting in ability groups 
ing, testings and guidance counseling which in turn 
foster racial segregation within the schools? 

161. Outcome Question ; Do vocational programs reduce 
radical political activity in a society insofar as 
they tend to integrate new and/or alienated segments 
of the population into the established mainstream of 
the nation's socioeconomic life? 

162* Outcome Question i Has one of the major mpacts of 
vocational programs been the continuous reinforce^ 
ment of class bias and prejudice among students in 
public education? 

163. Outcome Question s Does establietaient of vocational 
'programs result in easing unemployment for the few 
but in shifting the burden of finding work to other i 
less fortunate^ segments of the population? 

164* Outcome Question i Has vocational education provided 
institutions and citizens with an awareness of ^ in- 
formation about ^ and an understanding of the impli^ 
cations of poverty conditions within the community^ 
both socioeconomic and political? 

165. Outcome Question i Does the establistoent of a voca-^ 
tional program result in a heightended awareness 
within the community of the importance of developing 
human resources? 

166- Outcome Question i Has vocational education created 

inGreased support for general quality education with-- 
in communities because of the useful social services 
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it supplies such as providing needed manpower # inoul- 
catlng aacaptable social norms and values^ and creat- 
ing new job opportunities? 

167. Outcome Queation g Has vocational education been suc- 
cesaful in involving conmunity leaders in education 
through its use of citizen advisory conunitteee? 

168, Qutcome Question i Does vocational education support 
and reinforce the social r econoinic^ and political 
status quo? 

169 i Outcome Queation i Has one of the major impacts of 

vocational programs bean to reorient education toward 
vocational ends? 

170. Outcome Queation g Has vocational education contrib^ 
uted to the upgrading of teaching skills through its 
use of inservice training for vocational teachers? 

171. Ou tcome Queation i Does the establislunent of voca* 
tional-- technical schools and community collegs pro-^ 
vide more types of adult education to the community? 

17 2- Qutcome Question : Does the establishment of a voca* 
tional--technic^r college system result in an expan^ 
slon of the secondary vocational education system? 

173. Qutcome Question i Is vocational education^ through 
cooperation" with CETA, trade unions, business, in* 
dustry, and the military, effecting institutional 
changes in educational delivery systems? 

174, QutcQme Question i Have the innovative teaching meth-- 
ods in vocational education influenced educational 
practices in general^ e*g*f the establishment of 
open-gentry and open^exit programs, grouped ability 
classes, etc,? 

17 5. Qutcome Que stioni Does the separation characteristic 
of the relations between vocational and non^vocational 
or acad^iio students in training carry over to the 
relations in their teaching careers, thereby severely 
hindering conmiunication and cooperation between voca- 
tional and general/academic programs? 

17 6* Outcome Question : Has vocational education been an 
important factor in promoting geographic mobility 
within society by providing skills needed throughout 
the nation in such areas as computer programming. 
Welding f and secretarial services? 
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195 • Outcome Question i Does the eKistence of vocational 
programs in high schools foster a dual systCTi of 
education by tracking, lower-class , rainority boys 
into low status progrMas and in tracking lower-class, 
minority girls into occupations traditionally female 
on the one hand and on the other ^ by -tracking middle-- 
and upper-class students of both sexes into academic 
programs? 

196. Outcome Question i Do sex--stereotyping biases of 
vocational educators and counselors limit women and 
girls to training for only a narrow range of occu^ 
pations? 

197. Outcome Question i Has participation in vocational 
programs enabled many wonien to overcome their early 
socialisation and to obtain skills that qualify them 
for work in non-^traditional occupations? 

198. Outcome Question i Does participation in a vocational 
program prepare the displaced homemaker to become 
economically and psychologically self-sufficient, 
free from dependence on federal and state financial- 
assistance programs? 

199. Outcome Questioni Do the childcare facilities pro- 
vided by many adult vocational programs allow women 
formerly unable to afford training to acquire skills 
which enable them to earn good salaries? 

200. Outcome Question i Are vocational education programs 
likely to be successfully or unsuccessful with regard 
to program completion and/or job placement of the 
mentally retarded? 

201. Outcome Question i As a result of the curricula found 
^ in prisons, is the vocational training received by 

female offenders generally limited to the areas' of 
cosmetology and business? 

202. Outcome Question i Are the vocational facilities in 
prison so obsolescent, in such poor condition/ and 
of such poor quality that inmates cannot receive ad-- 
equate training to prepare for post^prison ffliployment? 

203. Outcome Question i By participation in prison voca-^ 
tlonal programs do inmates acquire skills and work 
experience only in a narrow occupational area or in 
an area relevant only to prison maintenance? 
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185* Outcome Question i Does participation in voGational 
training by the educationally and economically, dis- 
advantaged tend to reduce structural unemployment? 

186. OutcQine Question i Does vocational education promote 
the integration of the educationally and economically 
disadvantaged into society's occupational structure? 

187* Outcome Question i Does establishment of voaational 
programe" promote the development of business owner* 
ship by Blacks? 

188. Outcome Question : Does participation in vocational 
programa in large cities and in de facto segregated 
school systems cause Blacks and other minorities to 
receive training in areas requiring lower Bkille than 
in those areas of training provided by predominantly 
white schools? 

189. Outcome Question ! Do many vocational educato'rs im^ 
pose training in manual skills on Blacks and other 
minorities by establishing the curriculum so that no 
other choice or alternative is available? 

190. Outcome Question : a result of the stringent en- 
trance reguirernents imposed by many vocational pro- 
grams ^ are Blacks and other rninorlties sometimes ex* 
clouded from vocational programs? 

191. Outcome Question s Has participation of Blacks and 
other minority groups in vocational training been 
obstructed because the modern facilities are located 
in suburbs f outside of urban areas having a high con- 
centration of non^white population? 

19 2. Outcome Question s Does participation of Blacks in 

vocational programs limit their training to areas of 
traditonally -'Negro" employment ^ e.g . / occupations 
where Blacks provide services to other Blacks (as in 
barbering) . 

193, Outcome Question i Does vocational education contrib- 
ute to the increased representation of minorities 
among the leaders in the business community? 

194. Outcome Question i Has vocational education promoted 
opportunities for Blacks and other minorities to 
enter high paying occupations? 
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195 t Outcome Question i Does the existence of vocational 
programs in high schools foster a dual systati of 
education by tracking. lower-claBs, minority boys 
into low status progrMas and in tracking lower-class, 
minority girls into occupations traditionally female 
on the one hand and on the other ^ by -tracking middle-- 
and upper-class students of both aexes into academic 
programs? 

196. Outcome Question g Do sex-=stereotyping biases of 
vocational educators and counselors limit women and 
girls to training for only a narrow range of occu^ 
pations? 

197. Outcome Question i Has participation in vocational 
programs enabled many women to overcome their early 
socialisation and to obtain skills that qualify them 
for work in non-^traditional occupations? 

198. Outcome Question i Does participation in a vocational 
program prepare the displaaed homemaker to become 
economically and psychologically self-sufficient, 
free from dependence on federal and state financial- 
assistance programs? 

199. Outcome Questioni Do the childcare facilities pro- 
vided by many adult vocational programs allow women 
formerly unable to afford training to acquire skills 
which enable them to earn good salaries? 

200. Outcome Question i Are vocational education programs 
likely to be successfully or unsuccessful with regard 
to program completion and/or job placement of the 
mentally retarded? 

201. Outcome Question i As a result of the curricula found 
^ in prisons, is the vocational training received by 

female offenders generally limited to the areas^ of 
cosmetology and business? 

202. Outcome Question i Are the vocational facilities in 
prison so obsolescent, in such poor condition/ and 
of such poor quality that inmates cannot receive ad-- 
equate training to prepare for post^prison CTftployment? 

203* Outcome Ques tion i By participation in prison voca-^ 
tional programs do inmates acquire skills and work 
experience only in a narrow occupational area or in 
an area relevant only to prison maintenance? 
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204. Outaoine Question i Have voGational programs in prisons 
led to an increase in the incidence of ©nployment of 
offendare? 

205. Outcome Question i Does participation in vocational 
programs teach Native Americans on small reservations 
useful techniques for marketing their handicrafts and 
wares of reasonable financial profits? 

206. Outaome Queetion i To meet the needs of special groups, 
have vocational programs attempted to adapt the in^ 
structural material and facilities used in training 
regular students to make them effective in training 
the physically, emotionally, and mentally handicapped? 

207. Outcome Question i In their attempts to meet the needs 
of special groups* have vocational programs organized 
their curricula so that prograjn options offered regu- 
lar students are also offered to physically handioapped 
students? 

208. Outcome Question : Has the existence of vocational 
programs increased the participation of women in the 
labor force? 



C. OUTCOME QUESTI ONS HAV^ING SPECIAL NEEDS SUBPOPULATIONS (OR 
MEMBERS THEREOF) "THE "afTECTED ENTITY ; 

(2) RELATED PREDOMINANTLY TO NON-ECONOMIC ROLES 

209 i Outcome Queetion i As a result of participation in 
vocational programs, do minority group students de-- 
velop greater capability to assimilate and to adapt 
to general cultural patterns than similar groups 
participating in general/academic programs? 

210- Outcome Question : Has vocational education been 

effective in providing access to programs in general 
and in developing specific types of programs that 
help equalize the life chances of students with 
special needs? 

211. Outcome Question i Does the ongoing existence of a 

vocatlonal~~educ^tion system significantly contribute 
to the equality of educational opportunities for 
Blacks and other minority groups? 

212* Outcome Question i Where peer counselors and instruc-* 
tors have been recruited in vocational programs/ have 
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individuals deficient in motivation and aelf-image 
improved in these aspects? 

213 » Outcome Question i Is the success of minorities in 

vocational programs often undermined by diacriminatory , 
negative attitudes on the part of vocational educators 
and counselors? 

214* Outcome Ouestlon i Do those who differ culturally and 
linguistically from the main culture eKperience ' aeri- 
ous learning and psychological difficulties during 
training when vocational educators and counselors ex^ 
hibit insensitivity to these differences? 

215. Outcome Question ; Are minority students who compare 
unfavorably with their peera as a result of inappro^ 
priate testing often relegated to upgraded vocational 
centers or special education when with proper testing 
and counseling, they could progress favorably with 
their peer group? 

216* Outcome Question i Do tests in vocational programs 
reflecting Caucasian standards and cultures cause 
minority students (especially those of a linguistic 
minority) to score unfavorable without reflecting , 
the actual native ability of those tested? 

217. Outcome Question : Do many handicapped and/or dis^ 
advantaged students who have displayed various anti-- 
social behaviors become leas disruptive in the class- 
room as a result of their participation in vocational 
courses? 

218. Outcome Question : Do disadvantaged youths from a 

" we 1 f ar e =- e t h n i c " oriented home life acquire motiva-- 
tiona and attitudes associated with a "work-^ethic" 
culture due to their participation in vocational 
programs? 

219. Outoome Question ; Has vocational education aroused 
deep resentoent among many Blaeki who consider avail-^ 
able programs poor in quality and organised to pre-- 
pare Black youth for traditionally "Negro" jobs? 

220* Outcome question i Do vocational prograns that train 
members of special needs groups fail to teach these 
students how to deal effectively with the sooial 
pressures to be confronted in the work world (dia-* 
crimination^ rejection^ etc*)? 
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221* Outcome Question i Do teachers, counselors^ and ad- 
minietrators who tend to think in terms of "men's 
trades'* and "women's trades" transfer these percep- 
tions to vocational students? 

222, Outcome Question i Does segregation of sexes In voca- 
tional programs tend to perpetuate existing condi- 
tions of sex inequity with sex-role stereotyping? 

223, Outcome Q uestion s When vocational programs place 
the mentally" handicapped and the disadvantaged in 
the same courses/ do the disadvantaged students re- 
sent sharing classes with the mentally retarded? 

224, Outcome. Question i Are extracurricular activities 
associated with "vocational programs^ e*g./ youth 
organi mtionSf organised to allow full participation 
by physically handicapped students? 



courses ter deculturaliiation of Native Americans? 

226. Outcome Question i Does vocational education increase 
the quantity and quality of political skills, e,g,, 
gathering information to support legislation^ lobby'- 
ing/ voting/ etc./ among those who are concerned with 
programs for special groups, 

227. Outcome Question s Is it the case that because in 
most states certification for teaching vocational 
courses requires only a minimum number of hours of 
instruction (usually gained while the individual is 
already teaching vocational courses) / many teachers 
are not qualified to meet the unique problems of 
teaching disadvantaged and handicapped students? 

228. Outcome Question s Is the term "disadvantaged" ap-- 
plied by some administrators in such a way that it 
becomes all-encompaBsing/ thereby overlooking the 
needs of the actual disadvantaged population? 
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INDEX OF SPECIFIC TOPICS 

The outcome questions in the compendium (Section II) 
have been numbered for identification sequentially beginning 
with ^1 and extending to #228* Although it is hoped that 
most of the major topics relevant to vocational education are 
Included in the index ^ It is not exhauative with regard to th 
possible topics that could have been included. Lastly^ top- 
ics have been crois--ref erenced where it seemed advieable 
and/or appropriate. The major topic headings are arranged in 
alphabatioal order, followed by the identification numbers 
preceding each outcome question in the compendium. For exam-- 
pie, if the earnings of vocational students is the topic of 
interest, a reader would find "earnings" entered in the index 
in the following wayi 

Earnings, of vocational students, 3S, 39, 40^ 88^ 89, 90, 
91/ 92, . , . , 

The reader can then refer to the outcome questions having the 
identification numbers listed after the topic. We have not 
exhaustively listed every outcome question potentially re^ 
lated to a specific topic. In obvious cases this has been 
r.epresented by a series of periods following the last iden-- 
tification number listed* For example, given the large num- 
ber of questions dealing with "postseaondary " students, "post- 
secondary" has been entered as below i 

Postsecondary education/ vocational, 30/ 31, 32, 33, 34, 



The reader is encouraged to scan the list to insure that all 
outcome questions relevant to the topic of Interest have been 
noticed. 

In eome cases our sample of outcome questions lacked an 
entry for topics that are of concern to individuals inter- 
ested in vocational education. Whenever appropriate, we have 
included these topics in the index and referred the reader to 
outcome questions that could he formulated so as to be appli- 
cable to those topics. For example^ we have no outcome ques-- 
tions that specifically addresses vocational students who are 
migrants. Nevertheless, "migrants" is a topic included in 
the index in the following manner i 

Migrants, 209x, 213X/ 214x/ 215x, 216x, , , . . 

Outcome question #209x asks "Does participation in vocational 
programs develop within students who belong to minority groups 
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a greater capability to assimilate and to adapt to the gen- 
eral cultural patterns of the conununity * * . Although 
the question refers to minority students rather than to stu^ 
dents who are migrants, it could serve as an hypothesis about 
migrant students, as well. In this way we hope to generate 
some thought about possible outcomes that while not indludecl 
in our list of outcome questioni# may nevertheless be impor*- 
tant • 



Acadeinia, 

Achievement, 104, 213, 215, 216 
Administrators, vocational, 99, 144, 221, 228 

Admissions, 

to college, 27, 28 

to PoBtsecondary , vocational programs, 26, 28, 29, 190x 

to Vocational programs, 190 
Adults, in vocational education, 47x, 49k, 56k, 77k, 108k, 

218k .... 
Adult vocational training, 171, 180 
Alienation, 

from Self, 118 

from Society, 117, 161 

from Work, 116 

Benefits , financial 

of Postsecondary education, 38, 39, 40 

of vocational education to society, 141^ 145, 146, 149, 154 
of Vocational education to students, 3, 17, 110, 184 ^ 205 
Bilingual populations, see Minorities, ethnic and linguistic 
Blacks, 160, 187, 188, 189, 190, 191, 192, 210, 218, see Dis- 
advantaged, see Minorities 

Capital, 

Non-human, 146 
Childcare services, 199 
Citizens advisory council, 167 
Citizenship, 18, 138, 164 
College^ 

Admissions, see Admissions 

Attendance by vocational students, 28 
Communication skills , 127 
Community, 

Attitudes to vocational education, 166 

collages, 8, 30, 39, 41, 43, 44, 45, 171 

Involvement in vocational education, 167 
Competency, occupational , see 
Consumer, 

Benefits, 110, 178 

Education, 111 
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Cooperative vocational programs^ 11, 18, 91,184 
Correctional sub-populations^ 177, 201, 202, 203, 204 
Counseling # vocational, 7 2 

Counselors, vocational, 66, 212^ 213, 214, 221 

Decision making, 12, 13, 19, 119, 122, 123, 124, 125, 126, 136 
Disadvantaged populatione, educationally and economically, 

163, 164, 183, 185, 186, 211, 217, 
218, 222, 226, 227, see Blacks, 
see Handicapped^ see Minorities 

Discipline, 

in School, 217 

at Work, 50 
Discriminational see Stereotyping 
Drop-outs, from school, 22, 23 

Earnings, of vocational students, 38, 39, 40, 88, 89, 90, 91, 

92, 93, 183, 184, 198, 20S 

Economic , 

Benefits of vocational education, see Benefits 
Costs of vocational education, 15S, 156 
Education (al) , 

Attitudes of students toward, 2, 23, 24, 25, 30, 31, 32, 

33, 74, 101, 107, 219, 223 

Awareness, 166, 174 

Impact of vocational education on, 169, 171, 173, 174 
Quality of, 103 

Educators, vocational, 66, 99, 144, 189, 212, 213, 214, 221, 

227 

Elderly workers, see Older Workers 

Employers, 73, 87 

Employment, 

Status, 5, 6, 7, 8, 34, 35, 36, 96, 185 
Ex^of fenders, see Correctional subpopulations 

Family, 

Attitudes toward, 17, 128 



Geographic mobility, 61, 176 
Group , 

Tasks, 12, 23, 
Guidance, vocational, see Counseling, vocational 

Handicapped, 200, 206, 207, 217, 223, 224, 227, see Disad- 
vantaged, see Minorities 
High school curricula, 

Vocational, 1, 2, 3, 4, 5, 6, » . . * (refer to Compend- 
ium of Vocational Education Outcome Questions) 



Income/ see Earnings 

Industrial development, 146i- 147, 153 
Intereets / personal/ 106 

Interpersonal relatione, see Social skills 
Job (s) / 

Creation, 141. 142, 143, 144, 

Placement, 4, 75, 83, 200, 204 

Satisfaction, 9, 43, 44, 45, 46, 85, 86 

Seeking, 64, 75, 76, 77, 80, 81, 82, 83, 84 

Skills, see OcGupational skills 

Sucaess , 57 , 

Tenure, 77, 97 

Training-relatedness of, 1, 2, 32, 37, 51, 52, 53, 90, 
Juvenile delinquency, 21, 77 

Labor market benefits, see Benefits 
Leadership development, 12, 121 
Learning, 

Attitudes toward, 101, 

Problems with, 102, 214 
Leisure time, 19 

Life style and occiipational choice, 19, 134, 157 

Men, in non^traditional roles, 79x 
Migrants, 209k, 213k, 214k, 215k, 216k, , , , , 
Minoritiea, ethnic and linguistic, 181, 188, 189, 190, 191, 

193, 194, 195, 209. 213, 
214, 215, 216, see Blacks, 
see DiBadvantaged, see 
Handicapped 

Mothers and work, 198, 199 

Native ^ericans, 205, 225, see Disadvantaged, see Minorities 
Occupation (al) , 

Choice, 57, 58, 60, 61, 63, 64, 65, 66, 68, 69, 70, 72, 81, 
122, 181, 188, 189, 190, 192, 195, 196, 197, 201, 

203 

Information, see Occupational choice, see Occupational In^ 
telligence 

Intelligence, 55, 56, 64, 65, 67, 68, 74, 76, 77, 116, 131, 
135 

Mobility, 40, 41, 57, 60, 95, 

Skills, 47, 48, 49, 50, 53, 57, 58, 59, 61, 62, 71, 79, 
180> 188, 189, 197, 199, 203 
Offenders, see Correctional sub--populations 
OJT (on-the-^job training) , 50 
Older workers, ISO 
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Opportunity I 

Equality of, in education, 190| 210, 211 
Equality of^ in occupations # 194 

Parents, 100, 139, 140 
Participation f 

in Politics, 137, 226 

in Society, 14, 15, 16 
Parsonal development, 13, 14, 17, 18, 19, 105, 113, 114, 124, 

125, 126 

pQStseoondary education, vocational, 30, 31, 32, 33, 34, • » 

. , (refer to Compendium 
of Vocational Education 
Outcome Queations) 

Poverty, and vocational education, 21, 164 

Problem-solving, 123, 124 

Prison, 

Education in, see Correctional sub=populations 
Facilities, 202 

Residential vocational schools, 21 
Resources , 

Human, 98, 148, 150, 151, 152, 165 

Non^human, 112 
Rural vocational programs, 93, 2 23 

Safety habits, 54 
Secondary education, 

vocational, 1, 2, 3, 4, 5, , * . * (refer to Compendium of 
Vocational Education Outcome Questions) 
Self -employment, 187, 205 
Sex-equity, see Stereotyping, sex 

Self-image, 98, 104, 115, 117, 118, 129, 137, 212, 220, 
221 

Skills, 

CODinq, 11, 17, 18, 76, 

Entry-level, 47, 48, 180 

Oocupational, see Occupational skills 

Political, 226 

Social, see Social skills 

Transferable, work-related, 57, S9, 61 

Specific, work-related, 58, 60 
Social, 

Awareness, 14, 15, 16, 119, 120, 122, 164, 165 
Mobility, see Social status 

Skills, 11, 18, 106, 119, 120, 121, 123, 127, 132, 220 
Stability, 161, 168 ' 
Status, 10, 41, 42, 94, 95, 157 
Understandings, 14, 109, 116, 135, 136 
Special needs sub-populations , 161, 163, 181, 182, 220 226 
Blacks, see Blacks 
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Correational subpopulatione, see correctional subpopulationa 
Dim advantaged, see Disadvantaged 
Ex-offenders^ see Ex-o£fendare 
Handioapped, see Handicapped 
Migrants r see Migrants 

Minorities, see Minorities, ethnic and linguietic 
Native Americans, see Native Miericans 
Older workers, see Older workers 
Womeni see Women 
S t er eo ty p 1 ng , 1 3 0 

Clase, 158, 159, 162, 192k, 213x 

Sax, 160x, I62x, 192x, 195, 196, 213x, 221x, 222 
Race, 133, 160, 188, 192, 195, 213x 
Suburban voaational programs, 191 

Tests in school, 215, 216 
Transition, 

from Adolesence to adulthood, 17, 18, 20, 113 
to World-of-work, 18 

UnMiployment levels, see Employment status 
Urban vocational programs, 93 

Voaational, 

Administrators, see Administrators, vocational 
Education, 1, 2, 3, 4, 5, . . • * 
Educators, see Educators, vocational 
Fields, 

Agriculture, 10, 36, 42, 52, 89 

Distributive, 4, 45, 46, 94 

Health occupations, 8, 36, 51, 86 

Home economics, 62, 17 8 

Office occupations, 94 

Technical, 9, 36, 45, 94 

Trade and industry, 4, 9, 45 
Impact of vocational education, 170, 17 2, 17 5 
Placement services, 7 5 

Wages, see Earnings 

Women, 45, 79x, 195, 196, 197, 198, 199, 201, 208, 218, 221, 

222 

Work, 

Attitudes toward, 108, 131 

Habits, 50, 120, 131, 217, see Job Satisfaction 
Youth organisations, 11, 12, 13, 14, 15, 16, 18, 19, 224 
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EXAMINING VOCATIONAL EDUCATIOri O'lTnOMESJ 
PERSPECTIVE ON THE STATE OF THR ART" 



I, INTRODUCTION 



This paper deals with the theory and practice of voca- 
tional education outcomes evaluation, which is part of the 
broader field of educational evaluation* It is not intended 
as a review and synthesis paper or comprehensive state^of-- 
the-art essay on vocational education evaluation but mainly 
as a contribution to the conceptual basis of outcomes analysis, 

A. Purpose of the Essay 

The overall purpose of the essay and the study to which 
it relates is to help answer the research question i What 
particular outcomes are appropriate for use as criteria to 
evaluate vocational education programs? 

Special attention in this paper is focused on: 

— ^clarifying the nature and meaning of vocational educa- 
tion outeomes; 

--eKpanding awareness of the range and diversity of out- 
comes; 



This essay was prepared as part of The National Center * s 
applied research and development project, ^'Examining Voca- 
tional Education Outcomes and Their Correlates," under a con- 
tract with the Bureau of Occupational and Adult Education, 
Office of Education, U. Department of Health, Education, 
and Welfare. Other products of this study include an anno- 
tated bibliography of vocational education outcome studies 
(Attachment "A"' in the present report) and a thesaurus of 
vocational education outcome questions (Attachment "B" in 
the report) * The essay was written by Robert L, Darcy, who 
wishes to express appreciation to John T. Grasso, Michael 
Scriven, Douglas Sjogren, and Daniel L. Stuff lebeam for their 
review of, an earlier draft. The author also acknowledges, 
with special thanks, the helpful comments and suggestions re- 
ceived from staff colleagues and members of The National Cen- 
ter's evaluation technical advisory panel* 



321 



EKLC 



332 



c-2 



— deveioping an approach to outcomas avaluation within 
the general frMiework o£ vocational education evalua- 
tion; 

-^identifying some issues r problems, and practicee in the 
evaluation of vocational programs with reipect to out- 
comes; and 

— suggesting an agenda for strengthening research and 
evaluation in the field of vqcational education out-- 
comes, which in turn can contribute to Improving the 
effectiveness and efficiency of^ educational programs* 

The present paper does not identify a blue-ribbon list of 
valid, operational outcomes nor does it lay out a procedure 
for doing so. Those tasks remain to be accomplished in the 
future. • . 

The audience addressed by this paper includes educational 
researchers and evaluatorsy vocational education policy 
makers f academicr*^ and the general public* P^hile some of the 
ideas and information contained in this paper may be directly 
useful to practitioners, it is not intended as a how-to^do^it 
handbook for program managers, evaluatore, or classroom 
teachers.^ 

B* Rationale for Studying Educational Outcomas 

Outcomes analysis is important to observers of vocational 
education for at least three very practical reasons. First, 



To help meet the needs of practitioners^ The National 
Center has developed Evaluation Handbooks Guidelines and 
Practices for Follow-Up Studies of Former^ Vocational Education 
Students (Stephen J, Franchak and Janet Welskott, 1978) . The 
annotated bibliography alluded to in footnote 1 above reports 
on design and procedures that have been used in some 
afnpirical outcome studies conducted during the past decade. 
Another Center project completed in 1978 is Interpreting Out *- 
come Measures in Vocational Educations A Final' Report (Floyd 
McKinnay, Kenney 1- Gray, and Marie ^braml V Scheduled for 
publication by The National Center in 1979 are five state-of- 
the-art papers dealing with various aspects of evaluationi 
needs assessment^ longitudinal methods^ use of evaluative 
data, ijnpact evaluation, and data sources for vocational edu^ 
cation evaluation* In addition, the National Institute of 
Education is developing a series of background papers and 
evaluation studies on vocational education under a mandate 
included in the Education Amendments of 1976. 
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the etates are mandated by Congress to evaluate program ef- 
fectiveness on the basis of two specific outcome criteria i 
(a) emplo^ent in training-related occupations^ and (b) employ- 
er assessment of the training and preparation -for -employment 
received by students*^ Many more outcomes T such as educational 
achievement and human development^ are perceived by educators^ 
and others as being of iranense importance both to individual 
students and society. Second ^ public concern over tax bur- 
dens, rising costs of government, and accountability regard- 
ing returns from educational spending have led to a search for 
tangible indicators of program impact. Third, as educational 
evaluation has developed and matured, study of the broad area 
of outcomes and impact has assumed increased importance in a 
comprehensive program of evaluative research alongside such 
non-outcome variables as institutional context, student char-- 
acteristics, resources, program goals , and educational pro- 
cesses. 

Moreover, these practical considerations are in keeping 
with the fundamental purposes of scientific inquiry i to dis- 
cover (1) the consequences of given actions (including pro- 
grams, policies, and institutional arrangements) and (2) the 
specific actions that will produce given consequences* In 
the policy sciences, there is a third function of research, 
namely to generate information which can help decision makers 
select the goals that will be treated as desirable "given" 
consequences and provide evaluators with standards to be used 
in evaluating observed consequences. In the present context , 
we are addresBing conceptually, a question of "positive anal- 
ysis" — ^What are the' various possible consequences of vocational 
education programs? and a question of "normative analysis"— 
Which of these consequences are appropriate and feasible to 
be used as criteria for evaluating vocational programs?^ 



Education Amendments of 1976, P, L, 94-482, Sec* 112(b). 

4 . 

Nobel prize winner Milton Friedman quotes John Neville 

Keynes in distinguishing among "positive science," a system- 
izmd body of knowledge concerning what is; "nortnative science," 
knowledge addressing criteria of what ought to be; and "art," 
a system of rules for the attairaient of a given end. Essays 
in Positive Economics (Chicago: University of Chicago Press ^ 



19S3J, p. 3. 
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II* EDUCATIONAL EVALUATION AND OUTCOfffiS ANALYSIS 



A review of literature in the broad area of evaluationr 
the narrower field of educational avaluation^ and the more 
specimliged field of vocational education evaluation disclosee 
considerable variation in perceptions and preferenaes con- 
cerning the nature, purpose^ and mathods of assessment and 
evaluation - 



A. Asseesmenty Evaluati on and Research ■ \ 

In the Encyclopedia of Educational Evaluation , asseae- 
ment is defined as "the process of gathering data and fashion- 
ing thein into an interpretable fbrm."^ Worthen and Sanders 
in Educational Evaluationi Theory and Practice , identify five 
" schools of thought about how evaiuation ought to be defined * 
As practiced by their respective adherents , these approaches 
are characterized as: (1) "educational measurement," (2) 
"professional judgment," (3) "comparing performance data with 
clearly specified objectives," (4) "collecting information to 
assist decision-makers in choosing among available decision 
alternatives," and (5) "systematic collection and analysis 
of information to determine the worth of a thing." Irrespec- 
tive of approach, description is acknowledged to be a basic 
component of the evaluation process*® Evaluation is sometimes 



Sometimes the term "assessment" is used interchangeably :. 
with "evaluation" and with "measurement," See Scarvia Ander-* 
son and others, Encyclopedia of Educational Evaluation (San 
Franciscoi Jossey-Bass, 1975) p* 27, ~ " ~ 

^Blaine R, Worthen and James Sanders, Educational 
Evaluationi Theory and Practice (Worthington, Ohioi , Charles 
Jones, 1973) , pp* 20f , 109f , 125* In their treatment of de- 
scription as a component of formal evaluation, Worthen and 
Sanders focus on the ideas of Robert E, Stake. More recently, 
writing in the newsletter of The Evaluation Network, Joe B, 
Hansen emphasizes the importance of description in program 
evaluation and laments the bias which evaluators display for 
experimental studies to the neglect of full and accurate de- 
scription. "With the advent of modern statistical techniques 
and the wide availability of computers to perform statistical : 
analysis," hw writes, "educational research and evaluation 
have become associated with statistical methodology to the 
point of eclipsing more fundamental considerations t It could 
be interpreted by practitioners of this emerging art-science 
as a sign of insecurity — (footnote continued on next page) 
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referred to as evaluative (or evaluation) research . Scriven 
suggests that "the basic" distinction [between 'straight' re^ 
search and evaluation research] seems to be that evaluation 
research must produce as a Gonclusion exactly the kind of 
stateinent that social scientists have for years been taught 
is illegitimate: a judgment of valuer worth, or merit, "^^ In 
the present paper we are adopting the Worthen and Sanders/ 
Seriv^n definition of evaluation as the determination of the 
worth of a thing. In this view/ evaluation involves (a) ob- 
servation and description of what and (b) a value judg- 
ment concerning what should be ^ i.e., a judgment of merit or 
worth t ~ 

However defined/ evaluation has numerous purposes and 
methods. The former include aacountability , compliance ^ pro- 
ject nionitoring/ program improvement , and providing useful 
information for judging decision alternatives^ such as whether 
to continue/ expand, modify, or terminate programs.® Needs 
assassmsnts are conducted, arid both formative and summative 
evaluations are carried out* The diversity of evaluation 
methods is suggested by some of the terms one encounters in 
the literature: goal-based evaluation/ goal--free evaluation/ 
formal/ informal, objective/ subjective, process, on--site, 
product, follow-up, impact, payoff evaluation, benefit-cost 
studl®^/ cost^ef f ectlveneas analyses, and the sometimes amblg^ 
uous "program" evaluation. 

This kaleidescope of concepts, purposes, perceptions, and 
procadures adds a richness to evaluation that may, however, 
exact a toll. The other side of the diversity coin in educa- 
tional evaluation is a lack of consistency, clarity, and 



(continued) — that evaluators are often more concerned 
with the statistical significance of their results than with 
the Iniportanca of the questions to which the analyses are 
addrassad." See "The Formative^Suiraiative Dichotomy and the 
Role of Description in Program Evaluation," Evaluation News , 
No. 6/ August 19 78, pp , 14-16* 

7 

Michael Scriven/ "Evaluation Perspectives and Proced- 
ures/" pp* 3-93 in W* James Popham (edO Evaluation in Educa - 
tion (Berkeley I McCutchan, 1974), p* 4* 

8 _ ^ 

In Educational Evaluation and Decision Making (Itasca, 

111 pi Peacock, 1971) , prepared Jor the education society 
Phi Delta Kappa by Daniel l. Stufflebeam and others, evalua- 
tion Ib defined as "the process of delineating, obtaining/ 
and providing useful information for judging decision alter- 
natives*" 
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oredibility with respect to the knowledge base we have accumu- 
latsd in the field of aducational outcornes.^ 



B. The Education Enterprise and EducatlQnal Evaluation 

It will ba helpful to view educational outcomes and out'- 
comes evaluation in the broader framework of the overall edu^ 
cation enterprise. 



1976 report by the Committee on Vocational Education 
Research and Development (COVBRD) noted that "$250 million 
spent by the U. S, Office of Education in vocational education 
research and development during the last 10 years has not had 
documented^ widespread impact" ( Assessing Vocational Educatio n 
Research and Development # Nationai Academy or bciencea, p* i) • 
Commenting on COVBRD 's findings^ the president of the Ameri-^ 
can Vocational Education Research Association observed that 
"the substance and process of vocational education research 
leaves something to be desired" (J. Robert Warmbrodr "Evalua- 
tion Research in Vocational Education," paper presented at 
the AVERA annual meeting in Houston, TeKas, December 6, 1976 # 
PP* If 6^11.) Other evaluators have noted that "most of * , . 
[the evaluation of investments in the human agent, such as 
vocational and technical education] is of little or no use for 
purposes of making reasoned choices among competincf social 
progr^s . . . due to the methodological errors which mar 
many studies as well as the lack of effective techniques to 
handle many of the more difficult issues which arise as a part 
of the evaluative process (Jerome Moss, Jr., and Ernst W. 
Stromsdorfer , "Evaiuating Vocational and Technical Education 
Programs" in Gerald G. Somers and J* Kenneth Little, Vgca- 
tional Education; Today and Tomorrow , Center for Studies in 
Vocational and Technical Education, University of Wisconsin, 
1971, p. 221). Floyd L. McKinney found "a lack of consistency 
in the use of acceptable procedures in conducting follow-up 
studies" ( Program Evaluation in Vocational Educati on; A 
Review , ERlC~Cleari~nghouse on Career "Education, The Center 
for Vocational Education, Ohio State University, 1977, 

p, 24) . Writing in an official S. Department of Labor 
magazine in 1974, a Columbia university researcher observed 
that "the large body of studies tanaly^ing] the outcomes of 
high school vocational education [are] beset with conceptual 
problems, methodological pitfalls, and statistical limita- 
tions , . . , Proceeding from different values and assump- 
tions^ the analysts have not even agreed on the objectives or 
outcomes to be tested" (Beatrice Reubens, "Vocational Edu- 
cationi Performance and Potential," Manpower , July 1974, 
pp/ 23-'29) , For additional comments on the state of vocational 
education evaluation, see the Annotated Bibliography of Voca-* 
tional Education Outcome Studies cited in footnote 1 above* 
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Figure 1 lists six components of the educational syst^, 
each of which may be seen as an appropriate object for de^ 
scription and evaluation. These interdependent aets of edu-- 
Nation variables arei 

(1) ConteKt the milieu of circumstancee and needs # 
organizational arrangements^ societal values, insti-^ 
tutional settings, and the physical environment with- 
in which education takes place; 

(2) Students characteriatics of the potential learners, 
who are recipients of investment activities and whose 
human capital^^ will be enhanced when the investment 
activities are sucGessfully carried out; 

(3) Resources in money terms, the funds allocated to 
pay for real resources employed in educational pro- 
duction; in real terms, the quality and quantity of 
human and physical factors of production so CTiployed 
(e.g*, teachers, physical plant, instructional sup- 
plies and eguipment) ; 

(4) Goals the stated or unstated aims of educational 
programs (i,e., intended educational outcomes); and 

(5) Processes — » instructional methods, program proced- 
ures, and educational technologies (broadly defined) ; 

(6) Outcomes short-term consequences and longer-term 
impact resulting from vocational programs (i*e,, 
dependent variables associated with the interaation 
of the five preceding sets of independent variables) . 

This six^-faetor analytical framework is similar to the CIPP 
model (Context, I^nput/ Process, Product) described in evalu- 
ation literature, although the terms are not defined in 



Acquired capabilities that can be used in one's capac^ 
ity as a factor of production, i.e., as a human resource. 
See Robert L. Darcy, "The Nature of Economic Enterprise," in 
Edwin L* Harr (ed.) Vocational Guidance and Human Development 
(Boston: Houghton Mifflin, 1974), p." 121, ' Note tha^ human 
capital does not denote people/ but capabilities embodied in 
people. For a simplified explanation of human capital invest-- 
ment/ see William e, Becker, Jr., "Investment in Human Capi- 
tal," pp* 59^-76 in Warren G. Meyer (ed,) Vocational Education 
and the Nation's Economy (Washington, D. C . i ^erican Voca- 
tional Assocration, 1977) . 
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precisely the sarae way. It also resembles a conventional 
aducation production funGtion, which relates input factors 
to educational products or outputs. The five "input" fac- 
tors listed in Figure 1^ which' interact to conBtitute educa- 
tional progrmiis or " treatoients^ " are recast in Figure 2 (note 
input factors ni* * * ng) in the format of a production func- 
tion. This analogy suggests that over a period of time# and 
subject to the influence of intervening forces and error fac-- 
torSf it is theoretically possible to observe certain outcomes 
as having been generated by (i.e*/ significantly associated 
With) given educational treatmenti. 

As suggested above # each component of the educational 
enterprise represents a potential object of systematic inquiry 
and evaluation* Description and assessment may single out for 
particular scrutiny / student characteristics (e.g., age, sex, 
mental ability , socioeconomic background of vocational stu^ 
dents); or the quality of resources employed (e*g., training 
and experience of teachers); or educational outcomes (e.g., 
skill proficiency, employment status, hourly earnings) rather 
than addressing, under the rubric of "program evaluation," 
the totality of factors relevant to educational performance. 
Indeed, this selective , "partial equilibrium" methodology, 
based on the principles of specialization and division of 
labor, might produce the same results for vocational education 
evaluation that are claimed in economics i sharply increased 
productivity. We need to know far more than we presently know 
about student characteristics, context, resources, goals, and 
the processes of vocational education, as well as about out- 
comes* And certainly we need to know about correlations and 
interactions among the variables. The appeal for special 
attention to particular facets of the vocational education 
enterprise is not meant to imply that each component stands 
in isolation from the others. In education, as in the economic 
world, everything depends on everything else. Yet, it can 
still be fruitful to study parts and facets. 



See Daniel L. Stufflebeam, pp, 128-150 in Worthen and 
Sanders, Educational E valuation : Theory and Practice . 

See Richard V. Kauffman, A Study of the Educational 
Production Function , Ph.D, thesis, Departaient of Economics, 
Colorado State University, 1974, pp, 24-31 (reprinted in hard 
copy by University Microfilms, Ann Arbor, 1978); and Daniel C. 
Rogers and Hirsch S. Ruchlin, Economics and Educations Prin - 
ciples and Applications (New Yorki Free Press , 1971) ,""pp, 
107 £7 119ff , 
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THE EDUCATIONAL PRODUCTION FUNCTION AS A 
FRAMEWORK FOR EVALUATION 



f 0=£(S,C>R.G.P)* 

[Outcomes are a function of Students, Context, Rasources, Goals, Proceises 
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. Obsgrvid outconies are lubjict to the influence 
of ixtranaous forces and error factors , 



C, Vocational Education Outcomes 

The focus of this essay, however, is on vocational educa- 
tion outcomes, which are broadly defined as the consequences 
of vocational programs Figure 3 illustrates the range and 
divarsity of possible outcomes asaociated with vocational edu- 
cation programs. All outputs, products ^ consequences ^ effects 
results, and impact of vocational programs are reaognized as 
outcomes whether they are intended or unintended, positive or 
negative, short-^term or long-term, economic or noneconomic , 
direct or Indirect # These outcomes may affect individual stu- 
dents, society as a whole, particular societal entities (such 
as the local school system) ^ or functional characteristics 
such as the human resource base of the state and local economy 
The outcomes may be seen as resulting from a student's partic- 
ipation in vocational education or from the eKlitence o£ on-- 
going programs in the community , Outcomes are manifest as 
changes in individual or societal capabilities (including 
knowledge and skills) 9 attitudes, attributes, status, or cir-^ 
cumstances * 

Some outcomes, such as the acquisition of occupational 
skills by secondary school students who complete vocational 
programs, are objectively observable and readily quantifiable * 
Others, such as the development of work-related attitudes or 
leadership capability, may be perceived more subjectively 
while defying straight-forward empirical observationi Ease 
of measurement does not qualify a particular outcome as nec^ 
essarily important or valid as a criterion for evaluating 
vocational education, though it certainly may offer practical 
advantages for such purposes. 

Figure 3 does not purport to identify all the kinds and 
characteristics of educational outcomes * No eKplicit ref er^ 
enee, for example, is made to physical, esthetic, or moral 
development. The representation is suggestive and impression^ 
istic/ hence the question marks. Every effect that is 



Definitions of vocational education and vocational 
programs are presented in Section III below* 

14 

A distinction is sometimes made between educational 
outputs and outcomes, the latter implying change while the 
former merely suggests "through-put." No attempt is made in 
this paper to distinguish among such terms as educational pro- 
duct, output, outcome, impact, etc* As indicated in Fig- 
ure 3, outcomes are broadly defined to include all of the 
consequences and results of the education enterprise. 
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ILLUSTRATION 0^ SOME OF THE DIVERSE TYPEB 
AND CHAnACTERISTICS OF VOCATIONAL EDUCATION OUTCOf^ES 
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(AttainQd Goals, Side Effscts) 
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significantly linked with a vocatiorial program is regarded 
as a vocational education outcome. Infornial observation/ 
social criticism/ and educational research have made the 
American people keenly aware of how much we do not know 
about tha effects of schooling. We therefore need to apeak 
of possible outcomes of vocational programs without confin*^ 
ing the analysis to such narrow outcome sets as pre- 
announced goals ^ presumed benefits, or quantitatively 
measured impact . 

Outcomes typically are multldinnensional^ — e^g., a jniK-- 
tura of educational and economic consecfuenees, cognitive and 
affective^ individual and societal — and they occur as "joint 
products" in the sense of niultiple outputs occurring to-- 
gether in a iingle production process. Thus^ just as lumbej: 
and wood particles are simultaneous outputs of the logging 
industry, so too are occupational skills, job-related atti-^ 
tudas, and a high school diploma — all "products" resulting 
from the same schooling eKperience, Which is the "product" 
and which the "byproduct" is a matter of selective per- 
ception. A comprehensive asBessinent of vocational educa- 
tion outcomes would acknowledge all of the observable con- 
sequences and presumably focus on the ones which are of 
special relevance to the evaluation task at hand. It would 
also recognize the opportunity costs or tradeoffs associated 
with vocational programs, l.e, ^ the alternative educational 
outcomes foregone because the resources are being utilized 
in vocational programs* 

Differential outconies may be associated with variations 
in educational treatment (see Section III below) , intensity 
of treatment, i.e., "time on topic"), educational level 
(high school, post-secondary, adult), student oharacteristics 
(age, sex, race/ethnicity, socioeconomic background, physi-- 
cai or mental handicap, aptitudes , career interests, etc.) 
and a host of other variables indicated in Figures 1 and 2 
above. This follows from the recognition that variation in 
the *• input" factors of the educational production function 
(Figure 2 above) predictably will result in differential 
outcomes. 

Awareness of the broad range, diversity, and multi^ 
dimensional nature of possible educational outcomes is im- 
portant to practitioners, researcher s, evaluators^ policy 
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makers, and the general public. It helps offset the narrc^/ 
perGeptioh of equating outaomes simply with goals or benefits, 
while igiioring "side effects." This eHnancled \riew in turji 
triggers a search for evaluative standaroR heyond preanno-unc^cl 
program objectives and can stimulate a re examination off pro-- 
gram Implanentation strategies and overall policy, 15 

As an example^ when state and local units of governinent 
first enacted general sales taxes, their single purpose v^fis 
to raise revenue. Attention was focused on the outcome of 
fiscal productivity, Another outcome of the ta^c, however / 
was its regressive burdens low--incomG consumers paid higher 
effective tax rates {tmK payments as a parcentege of income) 
than high-income consumers. While the purpose of the ta>c 
was not to radistribute income, that was nevertheleas an 
outqofne. AwareneBs of this consequence eventually led inany 
states to amend the general sales tak by eKempting food^ 
thereby reducing the regressivity of the tax. 

Slinilarly, if an important outcome of vocational efluca- 
tion is to increase the rate of school completion even 
thoiigh the stated objective (or specified evaluation cri^ 
terion) is job placement^ then knowledge of the school^ 
completion outcome can be useful to policy makers as they 
revise legislation and allQcate resources for the support of 
various programs, "Side effects" are conseguences no less 
than attained objectives. As the environniental and consuiner 
protection fields illustrate, yesterday's side effect may be 
tomorrow's most significant outcome. 

Table 1 lists 3 0 questions concerning possible outcortas 
of vocational education. The outcome questions (verbal ex- 



In a recent monograph on noneconomic outcomes/ the 
S* Department of Labor noted the '■unfortunate tendency to 
evaluate many employment and training programs in terms of 
narrow economic ^ cost-benefit analyses alone ^" The Job Corps 
was cited as "a good example of a program that should not be 
viewed so narrowly---a program in which the noneconomic impacts 
are at least as important and possibly more so, than economic 
benefits such as higher earnings," The report identifies 21 
differant noneconomic outcomes in three areas i (1) job re-^ 
lated; (2) social^attitudinal; and (3) healths-related. The 
Outcomes are discussed in the context of '-social developitient 
objectives . , • rehabilitation goals^ and . , • remediation-" 
0, S* Department of Labor, The NQneconomio Imp acts of the 
Job Corps, R & n ^lonograph 64, Employment and Training Admin^ 
istration (Washington^ D.C*i U. S. Government Printing OEflce^ 
1978) pp* iil-'iv. Attachment "B" of the present report Iflen^ 
tifies a wide range of economic and noneconomic outcomes and 
discusses the various uses that can be made of a Gomprehensl\^Q 
list of outcome questions or hypotheses* 
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Table I 

POSSIBLE OUTCOMES OF VOCATIONAL IDUCATION t 
iLLUSTRAriON OF ^HE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES 


NO* _ Outcome Questions 


_Thinnatic _Area 


1* Do students become Qccupationaily proficient 
and ready for entry-level employment as a 
result o£ completing a vocational ptograni? 


Skill acquisition 


2* Wiat proportion of former vocationai stu^ 
derita are amployed in training-related jobs 
siK months following program coniplatiQn? 


Einployinent in 
training'-related jobs 


3* Are dropout xateo lower for vocational stii^ 
dents than for comparable students enrolled 
in genaral/academic curricula at the sec^ 
ondary school level? 


Schooling 


4^ Do vocational programs at all levels-^'^high 
school i poatsecandary , and adult-^-^sigriif i- 
cantly strengthen the human resource base of 
the local and state economy? 


Economic developrnent 


5, One year following program completion? are 
former voeational studenta earning higher 
hourly wages than coinparable, general/academiG 
studenta in similar jobs? 


Wage rates 


6. Are the parents of students who are enrolled 
in vocational programs better satisfied with 
the ©varall high school program than parents 
of atudants enrolled in general/academic 
curricula? 


Parental satisfaction 
with school program 


7, Do vocational students obtain jobs which they 
consider acceptable ^ whether in trainings 
related fields or not, within a specified 
time period following program completion? 


Employmerit 

• 


8, Are vocational programs more effective or 
less effective than general/aeademic cur-^ 
ricula in enhancing the self-image and self- 
confidence of students who are preparing for 
employment? 


Student self^iniaga 
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Table I (Continued) 

POSSIBLE OUTCOMES OF VOCATIONAL EDUCATION i 
ILLUSTRATION OF THE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES 


No_* OutQonie Questions 


Thematic Area 


9* Do vocational students receive higher emplDyer 
ratings than comparable general/academic stu= 
dents followiiig a specified period of employ- 
ment? 


Employer assessment 
of students 


10* Do Vocational students develop leadership 

ability through their participation in occu-- 
pationally^^related youth programs such as 
Distributive Education Clubs of America 
(DECA) / Futura Homeinakers of toiarica (FHA) , 
Future Farmers of toerica (FFA) , and VoGa= 
tional Industrial Clubs of America (VICA)? 


Leadership develops 
ment 


11 # Are students who are currently enrolled in 
vDcationaL programs better satisfied with 
their ov-arall school experience than compar- 
able students enrolled in general/academic 


Student satisfaction 
with schooling 


12 * Do graduates of vocational programs express 
inore, less^ or the same job satisfaction as 
cofliparable graduates of general/academic 
programs holding similar jobs? 


Job satisfaction 


13. Does participation in high school vocational 
programs reduce the incidence of school 
vandalisni and other antisocial behavior on 
the part of students? 


Antisocial behavior 


14. Does vocational education (high school ^ post^ 
secoJidary, adult) provide its graduates with 
a general set of skills and basic work dis^ 
ciplines that significantly increase their 
chances for successful employment? 


Bnploy abi 1 i ty 
development 



cpntinued _ -» 



C-17 



Table I (ContaLnued) 

POSSiaLE OUTCOMES Of VOCATIONAL EDUCATION? 
IIiLUSTRATION OF THE ^NGE AND DIVERSITY OF POSSIBLE OUTCOMES 


Nd* Outcome Questions 


Thematic Area 


15, By completing a voeational program? do stu- 
dante acquire job-seekirig skills such as how 
to write a resume, how to participate in an 
interview , and knowing v/here to look for job 
opportunities? 


Job^search skills 


16, Do students enrolled in vocational programs 

learn the basic cognitive slcills of apeakingi 
reading, writing^ ^nd numerical calculation 
as effectively as comparable students in 
acadamic/general curricula? 


Educational achieve-^ 
ment 


17, Does participation in a voeational program 
giva students a realistie undefstanding of 
what employees will eKpeet of them on the job? 


World-of-work 
understanding 


18, Is the occupational mobility of vaaational 
atudenta limited both horizontally and 
vertically by the content of the programs 
(m,g,f training in narrow slcill areas or in 
OGGupations with dilninlshing job opportuni- 
ties) ? 


Occupational mobility 


19* Does participation in a high school vocational 
program inhibit or enhance a student's oppor- 
tunities for a college education? 


College education 


20. Do students who complete vocational programs 
have a more realistic understanding of our 
society and its economic system than compare- 
able students who complete general/academic 
programs? 


SocioeconomiG 
understanding 


21* Are the returns from individual and social 
investments in vocational edtacation lower, 
higher i or the same as TRturns from 
investments made in ganfiral/academic pro-- 
grams below the baooalatireate level? 


Eeononiic returns 
to schooling 
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TaDle I icontinueaj 

POSSIBLE OUTCOMES OF VOCATIONAL EDUCATION i 
ILLSUT^TION OF THE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES 


jgo, outcoma Ouestions | Thematic Area 


22* Does participation in vocational progranis 

on the high school level contribiite measur- 
ably to the moral developinent of students? 


Moral development 


23^ Through their partiaipation in vocational 

education programs (e.g.^ cooperative educa= 
tion# placemant in part-'time joba) t do dis^ 
advantaged rninority youth earn more income 
while still in school than comparable students 
in general/academic programs? 


Student earnings 


24* Do high school^ postsecondary? and adult 
vocational programa increase occupational 
avmrenass^ training opportunitiea^ and job 
access for women in both traditional and 
nontraditional areas? 


Employment opportun- 
ities for women 


25* Does favorable community reaction to ongoing 
vocational programs result in general expres'^ 
sions of satisfaction and financial support 
for the entire local school system? 


Community support 
for education 


26- Do students who have participated in vocational 
programs on the high school level have more 
clearly defined occupational goals than those 
expressed by nonvocational students who enroll 
for postsecondary education? 


Career planning 


27, Do high school home economics courses slg- 
nificantly increase the consumer skills of 
students who have taken them? 


Consumer skills 
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Table I (Continued) 

POSSIBLE OUTCOMES OF VOCATIONAL EDUCATION s 
ILLUSTRATION OF THE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES 


NO. Outcome puestions 




28. Do cartain admissions and student assignment 
practiGas followed in vocational education 
tend to reduce or to reinforce occupational 
discrimination on the basis of race? clasSf 
or se5c? 


Occupational dis= 
crimination 


29. Is job-search time lower for students who 

participated in vocational prograinB, than for 
coniparable workers who did not? 


Unemploytnent 


30. How do the annual earnings of vocational grad^ 
uatea compare with those of general/academic 
graduates, both in the short term and in the 
long run? 


Earnings 
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on sueh topics as schooling^ antisocial behavior ^ ec^onomic 
development, parental attitudes^ and the like* Somu of the 
queatione imply positive consequenceB for voaationa;. pro^ 
gramsi while others may suggest negative or neutral effects. 
What is quite clear is that even a limited array of voca- 
tional education outcomes , as we have illuitrated hiire^ mx^ 
tends well beyond traditionally announced program objectives, 
labor market benefits to individual students, and aMegect 
harmful or beneficial effects on eduoatlonal development. 

Several of the outcome questions in Table I h^vm bf|n 
addressed in empirical research and evaluation studies* 
Undocumented aasertions have been made regarding thfj facts 
alluded to in other questions. And some of the queiitions 
have simply been posed without conjecture as to the answers. 
At this point in the discussion, in keeping with our intent 
to expand awareness of the range and diversity of oiitcomea, 
we simply wish to regard these questions as iilustraitive 
verbal expressions of possible results i waves and i^ipples 
that vocational programs may or may not be generating, 

D. Outcomes and Outcome Statements 

Figure 4 outlines a conceptual framework for evaluating 
educational outcomes. The outcomes themselves (Roman 
numeral I) are empirical phenomenal real-'world con^iequences 
of educational programs. Outcome Statements (11) sltb verbal 
expressions of these cause==and^e£fect phenomena, Eclucatlon^ 
al outcome statements can take the form of declaratJ^ve sen-- 
tences or hypotheses (as in Table I) * "Completers of high 
school vocational programs have lower rates of unemployment 
six months after graduation than comparable completeirs of 
academic/general curricula,'' Or they can be phrasecl as 
questions: ''Do completers of high school vocationa], pro-" 
grams have lower rates of unemployment six months ai^ter 
graduation than comparable completers of academic/geineral 
curricula?" An outcome statement describes what ha|)pens to 



For the identification of some of these hypot:heses 
plus annotations of empirical studies, see Attachmerit -'A" of 
this report* 

17 _ . . 

Attachment "B" lists 228 vocatronal education outcome 

questions, further illustrating the range and diverrdty of 

possible outcomes* The introduction to the thesaurvis also 

discusses the content of outcome statements and the rationale 

for using a question format in listing outcome verbiilizations , 
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whom and how, and in some cases also Indicates when^ where, 
and why. As a minimum^ a meaningful outcome statement 
clearly identifies (1) an outcome^ (2) the person or thing 
affected, and (3) the educational treatment* Often the 
statement (or outcome question) will also identify the 
agency through which the treatment is applied, a rationale 
for the posited outcome , and a time framework. Expressing 
an educational outcome in precise verbal form affords two 
important benefits i clarity of understanding and clarity of 
communication, hm Roget of Thesaurus fame and others have 
pointad out, words are not only a medium of communication, 
but equally important^ they are instruments of thought. 

In Table I, questions are used rather than statements 
or hypotheses. This is to avoid giving an impression that 
we are asserting beliefs or reporting documented facts. 

Outcome Measures (Point III in Figure 4) are empirical 
indicators of the educational outcomes. These indicators 
may be quantitative (differences in amount, eKpressed numeri 
cally/statistically) or qualitative (differences in kind^ 
described verbally) , The outcome measures are empirical 
indiQators of the changed conditions or circumstances re-^ 
suiting from an educational program* These changes may be 
observable direetly or indirectly* The measures may be 
bare descriptive indicators or evaluative indicators, Out^ 
come measures require empirical information such as employ-- 
mentment and wage data, and standards or norms for evaluat- 
ing the data* It is at point Ill-B fn the framework, there- 
fore^ that positive analysis of educational outcomes (ob-^ 
servation and description) merges with normative analysis 
(appraisal of data) to produce the judgment of worth that 
constitutes educational evaluation (IV) , 

The findings about program consequences (V) that result 
from evaluation research become available for a variety of 
applications including policy making, program improvement, 
accountability/ and decisions to terminate^ expand, or 
otherwise modify the education enterprise. Returning to 



The author is indebted, to Herbert A, Simon ^ winner of 
the 1978 Nobel Priie in Economics, for Roget 's observationi 
"The use of language is not confined to its being the medium 
through which we communicate our ideas to one another r it 
fulfills a no less important function as an instrument of 
thought; not being merely its vehicle, but giving it wings 
for flight." Herbert A, Simon and Allen Newell / "Models i 
Their Uses and Limitations" in Leonard D. White (editor) 
The State of the Social Sciences (Chicago i University of 
Chicago Press T 1955) / p. 66 / 
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EDUCATIOflAL OUTCOMES EVALUATION FRAIWORK 



(I) EDUCATIONAL OUTCOMES 




(11) OUTCOME STATEMENTS/HYPOTHESES/ QUESTIONS 




(III) OUTCOME MEASURES (INDICATORS) 




(III-A) EMPIRICAL DATA (QUALITY AMD ACCESSIBILITY) 




(III-B) STANDARDS FOR EVALUATING THE DATA 





(IV) ANALYSIS^ INTERPRETATION^ AND EVALUATION 




(V) FINDINGS CONCERNING THE CONSEQUENCES OF PARTICULAR PROGRAMS 




<VI) APPLICATIONS OF FINDINGS ABOUT OUTCOMES 

(VI-A) t POLICY (GOALS, RESOURCES, RESOURCE ALLOCATION) 

(VI-B) t PROGRAM DESIGN 

(VI-C) t PROGRAM MANAGEMENT 

ERIC (VI-D) •ACCOUNTABILITY g^y 
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III-B, we want to emphasize the point that introducing 
value eritsria (standards of goodness or badness) into the 
research process is prerequisite for makinq evaluative 
judgments. Whether a particular outcome such as the placa- 
n^ent of 6 Si of all vocational graduates in training -relatad 
jobs la considered good, bad, or indifferent depends on 
the evaluative standarda applied. Knowledge of facts alone 
may be aufflcient for research, but in the absence of a 
standard for comparison, there can be no meaningful 
evaluation. 

Standards or norms for evaluating observed outcomes can 
thus be regarded as the link between positive outcomes 
analysis and evaluation of outcomes, which by definition is 
a r -matlve undertaking. Where these standards come from 
and how they are used are therefore fundamental issues to 
be considered in this paper. 



III. SOME KEY ISSUES IN VOCATIONAL EDUCATION OUTCOMES EVALUATION 



As noted by the authors of a Rand Corporation report on 
the findings of educational research, "Education is an ex- 
tremely coniplex and subtle phenomenon." Moreover, educa- 
tional research is subject to serious limitations of data, 
measurement instruments, and experimental controls, includ- 
ing the influence ol factors outside the schools ("perhaps 
strong enough to 'swamp' the effects of variations in educa- 
tional practices"). The upshot is that "research has found 



Jerome Moss, Jr., The Evaluation of Occupational 
Education Programs, Technical Report, Research Coordination 
unit in Occupational Education, university of Minnesota, 
September 1968, p. 5. 

As suggested m Section i, outcomes analysis addresses 
two types of questions: What are the actual eonseguences of 
vocational education? and What are the desirable consequences 
of vocational education? The first is an empirrcal matter- 
of-fact question triggering a search for Inforniation about 
what is. The second is & normative question leading to a 
search for values indicating what ought to be . To the ex- 
tent that value judgments are influenced by evidence and 
reason, the search for values and facts will be intertwined. 
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nothing that consistently and unambiguously makes a differ- 
ence in student outcomes," 

While the Rand assessment dealt with general education 
(and in the conteKt of the federal govarnment ' § role in 
school finance) , the conclusions are not unlike those re- 
ported above for vocational education. Indeed ^ vocational 
education research and evaluation faces all of the general 
problems confronting education plus eome apecial difficul- 
ties of its own, to which attention is now turned. 



A. What is Vocational Education? 

The study of educational outcomes^ as illustrated in 
Figures 1 and 2 above # investigates the consequences of 
distinetive educational treatments or interventions , What 
is the nature of the treatment carrying the label "voca- 
tional education"? 

According to the Education Amendments of 1976 (Sec, 195) , 
the terjTi vocational education means "organised educational ^ 
programis which are directly related to the preparation of 
individuals for paid or unpaid employment^ or for additional 
preparation for a Gareer requiring other than a baccalaureate 
or advanced degree/" In further defining an organiaad edu- 
cation program, the law refers to ^^instruction related to 
the occupation or occupations for which students are in 
training or instruction necessary for students to benefit 
from such training," 

In a generic sense, vocational education can be defined 
as any program of education or training that enhances one's 
capability to pursue a vocation, i,e., a calling or occupa- 
tion or career. Defined more narrowly for programmatic 



Harvey A, Averch and others, How Ef fective Is School - 
in g? A Criti cal Review and Synthesis of Res earch FiniinqB , 
Rand Corporationr Santa Monica, calif., March 1972, See 
pp, ixf, 154, 160. . The authors emphasize that they are 
"not suggesting that nothing makes a difference, or that 
nothing 'works'; the problem is that rasearch findings 
oftan conflict* In a conversation with the author of this 
paper, Michael Scriven pointed out, however,, that Cognitive 
outcomes do consistently show up. It is when educators 
"slide into grandiosity" searching for impact on life style, 
income distribution, and similar variables that planned var- 
iations in education seem not to work. 
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purposes, vocational education refers to occupational pro- 
grams which are included in state plans developed by the re-- 
spective State Boards for Vocational Education* A similar 
administrative definition limits vocational education to 
programs supported by Federal funds. In practice, "the 
vocational education program has become so broad and diverse 
in its response to manpower demands and the needs of all 
ages and ethnic groups that it is almost impossible to de-^ 
scribe in any definitive way," 

One may question whether it makes sense to speak of 
"the" vocational education program, not only because of the 
manpower demands and needs alluded to, but for at least two 
additional reasons. It has frequently been observed that 
there is no national vocational education program in the 
United States. We have 50 different state programs each 
with its own supposed needs ^ resources, goals / priorities, 
standards f definitions , data requirements, and delivery 
systems* In light of these differences^ are there enough 
commonalities among state programs to discern a vocational 
education program that is reasonably "typical" for the 
nation? The answer would seem to be "Yes." Despite some 



Vocational education is defined in Encyclopedia 
Brittanica as ". , . instruction that is intende~d to fit 
persons for industrial or commercial occupations" (Vol. 23, 

95)' The Dictionary of Education (edited by CarFer^V. Good) 
defines vocational education as "a program of education below 
college grade organized to prepare the learner for entrance 
into a particular chosen vocation or to upgrade employed work- 
ers; , . , training or retraining . , . is given in schools or 
classes under public supervision and control or under r^ontract 
with a state board or local education agency" (p, S45) * In 
Ohio State University's Proposal for a National' Center for Be- 
search in Vocational Education, a distinction is noted between 
"Vocational Education (capitalized) , the program supported 
under federal legiElation; and vocational education in a gen- 
eric sense . . , [which] embraces all activities, public and 
private, which are designed to contribute directly to occupa- 
tional proficiency" (Vol, I, pp. If). EKamples of such activ- 
ities cited in the proposal include military training, Compre-- 
hensive Employment and Training Act (GETA) programs/ public and 
private school vocational programs, and apprenticeships. 



Mary L, Elli«, A Report to the Nation on VQca tional 
Education (College Park, Maryland^ Ellis AssociatesTTlTF) , 
p, 2. 
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interstate aifferences in Student characteristics (n in 
Figure 2 above) . and Context (n.) it seems reasonable^to this 
writer that nationally shared doals (n.) and the sKietence " 
of national markets for Resources (n J -and Processes • (n ) ^ 
would combine to establish vocational education proarami in 
the fifty states that are in essence quite similar. If this 
is m fact true, some important outcomes should be observable 
as a national pattern. , ww»eivdoj.e 

The other respect in which diversity might undermine 
the plausibility of treating vocational ediicatioh as an 
entity relates to differences among particular occupational 
programs. Does Distributive Education, for example, have 
less in common with Vocational Agriculture than it does with 
the acadeinic/general school curriculuni? Are. inter-program 
differences in goals, methods, personnel, and philosophy eo 
vast that the occupational curricula share little" other' than 
the "vocational" label? Is it true, as some have claimea, 
that "There are no vocational educators, only Vb Ag, Home Ec, 
and T & 1 educators"? Operating on the premise that the 
more specialized and unique a program is, the easier it is 
to demonstrate differential outcomes, a case can be made for 
evaluating highly specific occupational curricula (such as 
Air Conditioning, Automotive Services, Diesel Mechanics) 
rather than clusters (such as Trade and Industrial Occupa- 
tions) . However promising this "component-program" 
approach to evaluation might appear, it would nevertheless 
pose methodological problems of its own when trying to 
aggregate results for the purpose of producing generaliza- 
tions about "the outcomes of vocational education. " 

B. Vocational programs as Educational Treatments 

Answering the question, What is vocational education? 
is more than an "academic" exercise, if evaluation entails 
both description and appraisal as suggested in Section 2 
above, a necessary first step for evaluation is to identify 
the entity to be described. What are the essential features 
of the distinctive educational treatment or intervention be- 
ing evaluated? Figure 5 depicts the relationship between 
"input" variables and educational outcomes in a manner that 
differs somewhat from the production function of Figure 2. 
Intervention (or program treatment) is defined in Figure 5 
to Include Resources (n^). Goals (n ) , and Processes (n j ; 
while Students (n j) and' Context {n^) are depicted separitely. 
This formulation has the dual advantage of isolating the ,two 
"input" variables that are less amenable to variation while 
also highlighting the fact that outcomes may be observed 
with respect to either (a) individual students or (b) social 
context. The linkage on the left side of the schematic in- 
dicates continuous mutual interaction between Students and 



DETERMINANTS OF VOCATIOMAL EDUCATION OUT 
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ConteKt 
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*lntervention or educational treatment Is narrowly defined 
Rasourcae (n^) , Goals (n^) , and Processes (n ) . (See Pigu; 



'**Observable effaota of distinctive educational treatments 
inJluence q£ sxtraneous forces and error factors) ^ 



Figure 5 
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Context 



|;*lntervintion or educational treatment is narrowly difinsd here to ineowpasg ' 
^^.Risourcas (n^) , Goals {n^} , and Prociisis (ng). (Sei Pigure 2 abovij ^ 

fc.**Obsirvabli effects of distinctivi iducational ttsatments (subject to the 
|!-,influsnca of, extrantous forcis and error factors) * " , 



O 



Figure 5 

359 



C-29 



something of the magriitude of the Coleman Report," It might 
be conjectured that this strategy would also tend to remove 
the focused heat of educational accountability fro^ vocation- 
al education. 

Yet a legitimate question remains. So long ae schools 
offer a course or sat of courses and experiences under the 
rubric of vocational education—and with earmarked funding- 
there will be a feasonable eKpectatlon-fior^-an educational 
quid pro quo. What comes out of the vocational education 
process in reSpDhse to the specialized resources that go 
into It? Perhaps the outcomes that can reasonably be ex^ 
pected will be much more limited than occupational Gompetence , 
employability, employinent , successful work ad justirierit , and 
the like. Some differential outcoines^ however^ should pre- 
sumably be identifiable as the result of distinctive goals, 
processes, and resource utilization. 



Eva luative Cr I r.^r la. Experimental Controls, and Compariso n 
Groups " 

According to one of the definitions listed In Section 1^ 
evaluation is the process of comparing performanca data with 
clearly specified objectives. This conception of evaluation 
Is associated with the currently popult^r view that the sole 
or dominant purpose of evalufttlon It. prograni improvement ? 
"Evaluation 1,^ not to prove, but to improve," The approach 
helps explain why many ueople equate a study of vocational 
education outcomes with an investigation of goals and the 
extent to which they have or have not been achieved. Clear-- 



Warmbrod argues (on p, 9 of his BeacQn article) 
"When vocational aduc^tion is viewed as a function of the 
total school program/ BOme important Issues relevant to the 
design and conduct of evaluation studies come into focus." 
First, the target population becomes all of the school^ s 
students, not just vocational students. Second, the treats 
ments to be evaluated are various mixes of vocational and 
other courses, And third, '*an evaluation of the school's 
Occupational education goal becomes a part of the evaluation 
of all the goals and functions of the schools." 

26 _ 

Floyd L, McKlnney, Program Evaltiation in Vocational 

Eduf^atloni A Review (Columbus T The EBIC filearihghouse on 

Career Education , Center for Vocational Education , Ohio 

State University,, 1977) p. 2- 
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lyr however, the observable effects of vocational programg 
may differ from the preannounced intent of such programs not 
only in terms of discrepancies between stated outaome goals 
and actual achievement, but also with regard to such aide 
affects as (a) benefits not previously identified as goals 
and (b) other coriBequences that might be considered neutral 
or negative. Zn short, evaluation generically defined can 
serve other functions besides program improvement i and 
evaluative criteria can be derived from sources other than 
preannounced objectives. 

It was noted in the Introduction that one function of re- 
search is to describe the consequences of given actions, such 
as the observable outcomes of particular educational programs. 
Positive outcomas analyils, as defined above, addresses the 
£^11 r^nge of educational outcomes ^ not just traditional ob- 
jectives and prasumed benefits. This breadth of scope can 
produce insights w^hich in turn may generate new goals and 
priorities. In this way^ knowledge of what is can contribute 
to informed judgTOents of what ought to be. If a stated ob- 
jective of vocational education is preparation for employnient 
in a training-related occupation but an observed and perhaps 
unanticipated otitcome is employability in a much broader 
range of occupationB, with corresponding la^or market bene- 
fits for individual students^ then the latter goal might 
approprlateiy be added to or substituted for the narrower one. 

As we earlier emphasized, in moving from positive 
analysis to evaluation, there is a need for evaluative 
standards. Some of the standards may be goal-based; but 
lacking a comprehensive, omnlsoient statement of program ob- 
jectives^ 'l^ere atoes the full set of evaluation standards 
coma from? One source is professional judgment as to what 



It has been noted that an area "not well addressed 
in writings on the evaluation process is the establishment of 
Btandards and the relative utility of various types of 
Btandards (Quentln D* Clarkson and others, "A System for 
Establishing Evaluation Standards," CEDaF Quarterly, Vol, 10 
m. 1, Spring 1977, pp. 17-19). in their proposed classifica-^ 
tian system,, the authors distinguish four types of relative 
standards (Norinstive, rredlcted, Control, Trend) and two 
types of abSQlute criteria (Arbitrary or intuitive, and Value- 
based) . Moss and Stromsdorfer ("Evaluating Vocational and 
Teehnlcal Education Programs," p, 224) have observed that "the 
problefli of evaluating vocational education is confounded by 
the fact that its objectives and outputs are multidimensional 
. . , [and] succesBful development, weighting, and estimation 
of perforinanGe Inde^ces * , . has never been achieved," 
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constitutes reasonable and appropriate performance* Agree- 
inent ^ight be reached (by a panel of evaluators or their 
sponaors?) that an unemploynient rate under 10% constitutes 
an acceptable programmatie outconie. Or a flejcible norm could 
be eatabliahed for vocational graduates pegged at say 75% of 
the unemployment rate for all labor force participan^ts in the 
same age group* 

Evaluative standards can be derived from observing the 
outcomes of similar program activities, e.g., job»search 
time required for CETA graduates to find unsubsiai^ed jobs 
compared with job-search time for vocational school gradu- 
ates, or differences In outcomes resulting from the same 
program observed over a period of years. 

And, in certain instances, verbal absolutes such as 
•'satisfactory," '•high," nov/,*' '^good, " ''bad,'* can be posited 
as nonempirical standards. These seejn less likely to prove 
useful and credible than criteria baaed on empirical ob-- 
servations (whether quantitative or qualitative) . 

Perhaps the most promising source of evaluation stand- 
ards is needs assessment, vhich recognises that somecne ' s 
desiderata or "justified goals" --^though not necessarily the 
sponsor's or producer ' s--»are appropriate criteria for eval-- 
uating educational programs* 

Once again/ definitional problenis intrude on the issue 
of evaluative criteria. who are "vocational students," and 
who are the "nonvocational students" with whom status or 
performance might usefully he compared? A host of related 
issues arise, including the use of eKperimental controls and 
comparison groups. Definitions aside, is it appropriate to 
compare employment status and earnings of former vocational 
students v^ith nonvocational (i.e., acadeitiic and general) 
atudents? Should comparisons of labor market status and 
performance (e.g., finding employment in occupations related 
to training) only be made arnong specific occupational 
curricula within vocational education, or can such comparisons 
be made legitimately between the totality of vocationai stu-- 
aents on one hand versus all other students? If the latter, 
which particular eiitry-level jobs are "training-related" for 
graduates of the high school academic and general curricula? 

Consider the mandate under the Education Amendments of 
1976 to evaluate vocational programs '* according to the eKtent 
to Which program completers and leavers (a) find employment 
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in ocaupations related to their training, and (b) are con- 
sidered by their employers to be well-trained and prepared 
for eraployment * " What are the procedural options for carry-^ 
ihg o^t the evaluation? As an illustration^ without axten^ 
sive discuBSion, suppose the strategy is limited to identify^ 
ing the former vocational students who have found training^ 
related jobs (however defined) and then surveying their em^ 
ployers with given instruments* Assume that 60% of "program 
completers and leavers" found jobs in OGcupations related to 
their training and within this groups 80% were rated highly 
satisfactory with respect to their training and preparation 
for employrnent. How are the data to be evaluated? As noted 
in Section 11^ whether outcomes are conn idered good or bad 
depends on the comparison standards enplo7ed,29 

It was suggested above that In the absence of clearly 
defined performance objectives or predetermined evaluation 
standards^ it will be necessary to derive or develop such 
standards. One approach is to compare the status or per^ 
formance of vocational students with another group ^ similar 
in all important respects-^-age^ se^c, mental ability^ socio^ 
economic background^ etc»^^eKcept that the vocational students 
received a distinctive .educational treatment that was not 
provided to the comparison group*. In the case of secondary 
school students, the comparison might be made for vocational 
students versus nonvocational (i*e*^ academic plus general) 
who are similar in all characteristics except the treatment 
variable (participation in a vocational program) • Given such 
a comparison^ is it reasonable to hypothesize significant 
differences in the outcome variable for the respective groups? 

The general rationale for the hypothesis is that 
substantially different educational treatments can be expect^ 
ed to yield significantly different outcomes. Whether the 
differences between "vocational education" and the "regular 
treatment" can be eKpected to result in employability differ^ 
encas is a more specific and problematic question* Of the 
two outcome variables mentioned^ placement in a training^ 



It might be argued that interpretation and evaluation 
of the facts can be left to others-^managers, policy makers , 
interested parties^^who can furnish their own judgmental 
criteria* The question of who does the evaluating^ however, 
does not diminish the importance of how it is done. Procedures 
and value premises will be of concern to all groups having a 
stake in the evaluation, and whoever su:^^;!^lies the values will 
presumably be held accountable. Dogmatic standards and vote^ 
counting may seem expedient ? but that leaves important value 
questions unanswered * 
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related job Is a highly questionable evaluation criterion for 
the following reasoned (a) "relatedness" is ambiguous; 

(b) vocational students who were enrolled in a particular 
curriculuni may not deiire employment in a related occupation 

(eithar because their preferred option was not offered and 
they simply took something that was available^ or because ' 
the option they followed provided a useful opportunity to 
explore and reject a particular occupational field) ; 

(c) there are no appropriate comparison groups for whom 
differentiated fields of training can be identified; and (d) 
student access to particular jobs may have resulted more from 
personal contacts made by vocational teachers than from the 
possassion of occupational skills , 

It might nevertheless be appropriate to enumerate the 
former nonvocational students in the same school class who 
found jobs (irrespective of occupational fields) , draw a 
sample matched to the vocational students according to 
charaGtaristicB such as mental ability and socioeconomic 
background, and then compare employer assessments of voca-- 
tional and nonvocational students relative to the extent to 
which thay are considered ''well-trained and prepared for 
employment* " 

A third enumeration could also be made--of vocational 
students who found employment in occupations outside their 
field of training'--»f or comparison with the other two 
groups* 

If the intent were to test the efficacy of the dis-- 
tinctive vocational education treatment with respect to 
employabili ty (as measured by employer assessment of training 
and preparation) , it would seem plausible to expect lower 
rates of unemployment for students exposed to an employment" 
related vocational education treatment as compared with 
students who were not.^° If no significant differences occur 



Such expectations are plausible for the same reason 
one would expect a student who completed thr6e years of 
French to enjoy an advantage in the practice of that language 
over a comparable student who had spent three years taking 
history instead. A separate question relates to the amount 
of the differential in unemployment rates. Is achieving the 
differential worth whatever extra costs ^ if any, have been 
incurred in providing the vocational education program? 
Efficiency issues involving calculations of marginal benefits 
and marginal costs are not addressed in this paper, for two 

f continued on next oaael 



between the matched groups # or if the regular group performed 
better than the treatment group ^ this certainly would not 
demonetrate that vocational education was an abysmal failure. 
It would suggests however^ that success criteria might be 
sought elsewhere; perhaps vocational programs do not enhance 
employability differentially but have other worthwhile outcomaai' 
Alternative tests can presumably be designed to measure these du 
comes ^ thereby helping to disclose the real nature of the voaa- ; 
tional education outcomes function^ l.e*^ the matrix of conse- 
quences associated with this distinctive educational treatment* 

p. Nature and Quality of Outcome Data 

Nuinerous studies have commented on the avaiiability and | 
Interpretability of vocational education data, Evaluators 
have lamented the lack of national d^ta^ unanibiguous datap . 
accurate dat|£ comparable data^ "hard" data^ cost data^ and 
impact data. Our own review of empirical outcome studies and;/i 



^ (continued) reasons. First, until we know more 
about gross outcome variables^ it seems pointless if not par- ■■■I 
nicious to go through the aKercise of manipulating the finely 
tuned pecuniary values needed to generate benaf it-cofet esti- 
mates. Second, there is so much methodological discretion 
inherent in performing benefit-cost i rate--of-return, and 1 
cost'-ef fectiveness analyses that the findings may reflect 
the tastes and style of the investigator more faithfully 
than the nature of the empirical evidence* 

^^Among the data-related problems identified by McKinney^ | 
Gray, and Abram ("Interpreting Outcome Measures in Vocational 4 
Educationi A Final Report," p, 1) i are those of "definitions^ :| 
coimiunication, and problems of handling data,-' In the same :| 
report (Ch* III)^ Donald Drewes comments on data format and [< 
other issues associated with setting up the proposed Vocational;! 
Education Data System (VEDS) . For additional perspective on 
data problems, see Arthur M. Lee "Project Baselinei Historio-- J 
graphin Foundations for Vocational Education Statistics," Edu- :^ 
cational Researcher , Vol. 6/ No. 7 ( July-^August 1977) , pp, 3- A| 
9* As indicated in footnote 2 above/ two state-of-the-art S 
papers dealing with vocational education data are scheduled :| 
for publication by The National Center in 197 9. See Attachment ^ 
"A" of the present report for comments on data used in some rs'* 
cent studies of vocational education outcomes. 



related inquiries concerning the credibility and persuasive- 
nesB of outconie data suggest the need for a profile of data 
types classified on the basis of the nature of the data and 
hew they are generated. Table II distinguishes five types c 
data used in describing and appraising the consequenoes of 
vocational education* 

The literature review to which we allude disclosed 
heavier use of Status and Personal judgment data than the 
other types* In part, this reflects concern with the out- 
comes specified as evaluation criteria in the Education 
Amendments of 1976, namely "the eKtent to which [vocational] 
program completers and leavers • . , f Ind employment in 
occupations related to their training, and , . * are con- 
siderad by their employers to be well-trained and prepared 
for employment" (Sec. 112) . These particular outcome 
measures call for Status data and Personal Judgment data, 
respectively. Certain kinds of data are clearly more appro- 
priate than other types as empirical indicators of particula 
outcomes (see Figure 4 above) , In fact, outcome hypotheses 
can be formulated so explicitly that particular kinds of dat 
and specific evaluation standards are indicated very precise 
ly for use in performing the evaluation task* 

Additional factors influencing the types of outcome 
data used in evaluation include ease, timing, and costs of 
collection; credibility; comparability with other available 
data; amenability to statistical treatment and analysis; 
potential impact; and the ingenuity and skill of evaluation 
designers and data gatherers. Inevitably, there are trade- 
offs between such considerations as the ease and economy of 
collecting softer data of the Personal Judgment type and 
"hard" Monetary^ Status, or Performance data, which might 
have higher potential impact on certain audiences. 

Further attention will be given to these data-related 
issues in the second year of the National Center study, 
especially as efforts are made to develop operational pro- 
cedures for particular outcome measures* 



rV* SUMMARY AND RECOMMENDATIOnS 



AS an "instrumental" contribution to answering the 
research question "What particular outcomes are appropriate 
for use as criteria to evaluate vocational education pro-- 
grams?", this paper has attempted to identify and clarify 



= TYPES OF DATA USED IN VO»TI0NMj EDUCMION OUTCOMES EVALUATION 

P'V . . _ _ ' . _ __. _ ._ _. ^ 



kType of Data 


Nature off 
Observation 


Method of Observation 




Sr'-^ ■ - -■ 

Status 

II:'-'- 


Objective 


Observation and reporting of 
Gircumstance by a second par- 
ty or directly by respondent* 


Labor force 
tional stud 

School enrol 
dents curre 
enrolled in 


tonetary 

is --A- ■ ' 

.!';.V-'. = 


Objective 


Observation by a,, second party 
of Gircuinstanees reportable 
in money tenns* 


Hourly wages 
and nonvooa 
Annual earni 
Cost and ben 
progreans* 


Personal Judg= 
•ment 

i--"' ■ 

r'. 


Sub j active 


EKpressed directly by 
respondent. 


student opin 
conGerning 
of vocation 

Qaiployer ass 
to whieh st 
and prepare 


kscriptive 


Objectiva 


Characterization by a second 
party based on recordSf test 
results, and/or observation 
and professional assessment* 


Work-relattd 
butes of st 

Vocational i 
tions of st 


Performance 

It. : 


Objectiva 


Written or other tests of cog-- 
nitive and psyehomotor Jcnow- 
ledge and skills. 


Score on ski 
ment test. 

Score on rea« 
test. 



Tabli II 

mm Of DATA USED IN VO(^TIONAL EDUCATION OUTCOMlS EVALUATION 



ata 



Nature of 
Obsirvatlon 



Objective 



Objective 



Method of Observation 



Qbsirvation and reporting of 
circumstance by a second par- 
ty or directly by respondent* 



Observation by a, second party 
of circumstances reportable 
in menay tems. 



Examples 



Labor force statui of tomer voet-;; 
tional students. 

School inrollnient status of itu- ' 
dents currently or fomsrly 
enrolled in vocational prograini. 



Sub j estiva 



Objective 



Ejipressed directly by 
rispondent. 



CharacteriEation by a iicond 
party based on records, test 
results, and/or observation 
and professional asiesinient* 



Hourly wages of former vocational I 
and nonvoeational studinti. ■ = 

Annual iarningi of iormir itudihti.; 

Cost and binefits of vocational 
programs, ■ '? 



Student opinions or perceptioni. 

concerninf pality and relevance - ■ 

of vocational programs , . 
aiployer asiisitnint of the extint"..,:: 

to which studints are well trainsd: 

and prepared for ^plopent. 

Work-related attitt'^is and attri- 
butes of studiiits. 

Vocational intiriSts and aspira- 
tions of studifits. 



Objective 



Written or other testa e£ cog' 
nitive and paychoiotor know- 
ledge and sHills. 



Scora on skill proflciincy achiive-' 
ment test. j 

Score on readinf/stiath achieviaent'. |_ 
test, 
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standards in order to c&viy systematic inquiry beyond educa-.^ 
tional research into the sphere of c /aluation proper. Figurei 
4 also listed four potential areas in which evaluation find- -I 
ings might be applied— policy , program design, program ma nag ©J 
ment, and aGcountability— whether for prograni improvernent or 
other purposes. The importance of formulating outcorne state-rll 
ments explicitly and unambiguously was emphasizea both for 
clarity of thinking and communication. An outconie statement or 
question typically identifies an educational treatments af fast 
ed entity, and specif ic outcome as well as the ageincy through| 
which the treatment is delivered. An outcome statement may 
also identify a time framework and rationale for the stated 
outcome* 

Among the key issues identified in the area of outcomes";^ 
evaluation were the definition of vocational eduoation, the 
specification of vocational programs as educational treat- 
ments (see Figure S) , sources and usee of evaluative cri- 
teria ^ eKperimental controls? and the use of oonipariion 
groups in evaluating vocational education programs with 
respect to specified outcomee. Table 11 distinguished five 
types of data used in outcomes evaluation, classified 
according to the nature of the observation and methods of 
data collection, A review of outcome studies diselosed 
heavier use of Status and Personal Judgment data as opposed 
to Monetary, Ascriptive, and Performance data, with corre- 
sponding implications for generalizabillty , credibility, 
and impact on potential audiences, 

B. Sug geated Agenda for Improving Outcomes Evalr jation 



Paramount among the recommendations we can oifer for 
improving the art of outcomes evaluation is the formulation, 
of a research and development agenda to be conscientloualy 
followed over a period of say five years aimed at producing 
alear, accurate, relevant, credible, and communicable find^ fi 
ings. Salient features of such an agenda would be con- -jf 
ceptual and empirical answers to such questions as; 'I 

-1 

What is vocational education? . 



(2) Who are vocational fitudents? 



..,.-M 

3 



(3) What are the outcome goals of vocational j 
education? 4 



(4) What outcomes should be used as criteria to 
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©valuate vocational education, and why?3 2 

(5) What data are needed to describe and assess the 
effects at vocational education from the national 
as wmll as state and local perspectives^ and how 
can '/iUch data be best collected* and analyzed? 

(6) twiat standards should be applied to particular 
outcome measures for purposes of evaluation, 
and how should they be determined? 

(7) What specific "input" variables (under the head-- 
ings of Student Characteristics, Educational and 
Social Context, Resources, Goals, and Processes) 
in the vocational education enterprise are signi-- 
ficantly c. ;sociated with particular Outcomes^ and 
how are they functionally related? 

18) To what extent Jo extraneous forces and error 
factors inhibit the production of knowledge 
concerning "the correlates" of particular voca-^ 
tional education outcomes? 

(9) What evaluation methods and procedures should be 
used to produce accurate and credible generaliza= 
tions at the national level regarding the outcomes 
of vocational education? 

(10) How can evaluation findings on vocational educa- 
tion outcomes best be communicated to relevant 
audiences^ including policy makers^ practitioners^ 
and other evaluators? 

The conceptual and empirical answers prescribed for these 
questions must be meaningful and well documented so they can 
serve as a ba.iis for developing authoritative ^ widely shared 
definitions and standards. Where value criteria are* lacking 
and where empiriual evidence is incoMnlete and conceptualizations 



^^This is the research question currently being ad- 
dressed in the study to which this essay contributes. We 
feel that in its first year, this multi-year project pro- 
gressed about 35%-45% of the way toward developing some 
meaningful and useful answers to the question. During Year 
Two (January 1979 to January 1980) , attention will be 
focused on approximately six to eight specific outcomes-- 
deeigning operational procedures and determining the extent 
to which the selected outcomes and procedures are regarded 
by relevant audiences as appropriate for use in evaluating 
vocational programs * 
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imprecise r defiaiancies should be openly acknov/ledged and 
cogent rationale provided to justify whatever course of a 
is adopted as an eKpedient in light of the specified rati 
empirical^normative gaps - 

Action on the reconmended R&D agenda requires a wi 
ingness to acknowledge^-with neither deffensiveness nor re 
nation — ^©Kisting shortcomings as well as the strengthB of 
vocational education^ both with respect to program perfoi 
ance and the knovrledge we possess about the system's coir 
ponents and impact. And the "high tolerance for ambiguit 
that is sometimes counseled for vocational education re-- 
searchers and evaluators needs to be challenged in the im 
ests of vocational education's commitment to quality and 
sponsiveness . 
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